Drop The Pilot?

Yesterday's post was meant to be about the upcoming release of Tarisland. As you'll know, if you read it, that isn't quite how it turned out. Things began to spiral in the second paragraph and didn't begin to straighten out again until close to the end. 

As I improvised around the topic of digital ownership, I threw in a number of quite specific references to Spotify. Some of those came from my own memories of news items I'd read this year. Google Search provided the links. Others were found for me by Microsoft Copilot.

I only had the idea of using Copilot because I'd seen someone else do it first. Noizy Gamer posted a detailed article entitled "Using AI To Examine CIG Reaching $700 Million In Revenue" which I found interesting, not so much for what it said about Chris Roberts' almost superhuman ability to keep the Star Citizen funding coming but more for what it implied about the current state of so-called Artificial Intelligence in the context of research.

On a number of occasions, writing here about AI, I've said that one thing I'd really like from the new tech would be an unpaid, virtual research assistant. I have been known to engage, albeit sporadically and ineffectively, in some half-assed, amateur investigative journalism on the blog, mostly involving games and who's really making them - or making bank on them. Those posts always involve a lot of googling and skimming through tedious company websites, which isn't always the most thrilling way to spend an afternoon.

I wouldn't say I don't enjoy it because I do like doing research but it would be very handy to be able to hand off the more brain-numbing grunt work to an AI. And I have tried but until now the results have been so unreliable I've had to check every claim before I used it, meaning I might just as well have done the leg-work myself.

Noizy seemed pretty satisfied with the results Copilot came up with, though, and he's orders of magnitude more experienced in this kind of research than I am; much more qualified to judge how useful and trustworthy the information the AIs are returning from his requests. That thought came into my mind as I found myself somewhat unexpectedly extemporizing on things I really knew very little about in the opening paragraphs of yesterday's post. 

I'd already realized that if I was going to keep making sweeping statements about Spotify, I'd have to provide some evidence for them. Why not combine finding the information with a test run for Copilot? So that's what I did.

First, I had to find Copilot itself. I like to avoid getting locked up in Microsoft's ecosystem whenever possible so I don't usually use a Microsoft browser or a Microsoft account, although I have both should I need them. For this, I didn't bother with either. I just googled "Copilot" and took the first link. I did find it mildly ironic that I went immediately to Google to find a Microsoft product but not until after I'd already done it.

Once I had Copilot onscreen I found out there are effectively three versions available. You can pay £19 a month for a subscription to use Copilot Pro, which seems like a lot to me. There is a free month's trial at least, so you can test it out and see if you think it's worth paying that much. 

Alternatively, you can log into your Microsoft account for free but more limited access to much the same suite of services. Or you can do what I did and just ask questions without logging in to anything. I was quite surprised that was allowed. 

The free, unregistered version turned out to be fine for what I wanted. I only had to ask one question and a follow-up to get all the information I needed for the post. I simply asked Copilot to give me all the recent news stories involving Spotify and when the results came back looking like they'd been put together by Spotify's Head of PR, I rephrased the query to ask for all the news that had received a negative reception. That did the job.

When you ask Copilot a question, you can set three "conversational styles" for the output: Creative, Balanced or Precise. I wondered what effect that might have on accuracy but in fact it just changes how the AI "talks" to you, not the content of its replies. Or at least that's how Copilot says it works...

I picked Precise and got a nice, polite, formal set of answers. I like my research assistants to be professional, not familiar. I hadn't specified a number of results but both times I asked I got four news stories back and not the same ones each time, which makes sense since I'd changed the paramaters.

The results looked perfectly convincing but don't they always when you're dealing with AIs? By far the best part of the experience was that all of them came with citations and hyperlinks to the source material. 

There's a warning right on screen when you use Copilot, telling you the output is provided by AI and should be checked, which was the case back when it was still Bing, only then you had to go look at the tiny footnotes to see the sources. Copilot streamlines the process so all you need to do is click the links in the results themselves to check how accurate the data-gathering has been.

And it was entirely correct. All the links went to appropriate sources, which I was then able to reference in my post. It was nothing I couldn't have done with a Google Search but it was faster and easier. 

In fact, as far as I can tell, it was a Google search. Or, I guess, a Bing search, since this is Microsoft we're talking about. The AI framed the results with a polite introduction and sign-off and presented them neatly on the page with summaries, which certainly made it quick and easy for me but in essence it was pretty much the same set of links I could have pulled off the first page of Google.

This morning I asked Copilot to give me "a chronological list of superhero movies from the 1960s to the present day". I have it in mind to write a post about the CW adaptations of various DC heroes and whether their meandering, discursive approach fits that publisher's traditions and history better than big MCU movies have served Marvel, which is one hell of an undertaking and one that requires a lot more research than I can be bothered doing at the moment.

Copilot's response was less comprehensive than I expected, albeit still quite useful. I was hoping for a neat list of movies, suitable for cutting and pasting straight into a post. What I got was nothing much more than a gussied-up set of search results, complete with links, citing some very obvious, high-level websites - Letterboxd, Wikipedia, IMDB - plus one that felt less familiar but right on the money, the Superhero Database. 

It felt so much like a Google search thought I'd run a comparison check. I pasted the same enquiry into Google and got the same three, well-known names in the first four results, with a blog called Fights, Tights and Movie Nights filling in for the Superhero Database. 

I'm not drawing any conclusions from this minimal experimentation and I'm certainly not casting aspersions at Copilot (Which used just to be Bing AI.) or AIs in general. I don't know if Copilot's direct competitor, Google's Genesis (Which used to be Bard.) or ChatGPT (Which is still plain old ChatGPT and which also powers Copilot, just to make things even more murky.) could or would do better.

If you're interested in finding out, I refer you to the New Scientist article "Battle of the AI bots: Copilot vs ChatGPT vs Gemini", which I found by the simple expedient of typing "which is the best out of Copilot, Gemini and ChatGPT?" into Google. (Spoiler - "they’re all fairly matched"). Or you could ask the AIs themselves, I guess. I asked Copilot and it broadly agreed with New Scientist, concluding "the best choice depends on your specific requirements".

What a bunch of fence-sitters! And how insufferably polite!

My own feeling is that, for my particular purposes, I can't yet see much difference between what any of them is offering and just running the same query through Google. In the end, what I'm usually after is links, anyway. I need documented evidence to stand up whatever point I'm trying to make and for that I need to be able to cite a source. For that, I don't actually need a research assistant, just a gofer. By using AI to do my searches for me I'm basically hiring a Lamborghini to take me to the corner store.

What I do feel has changed is the degree of confidence I'm willing to invest in AI to shortcut some of the more tedious aspects of doing research. While I could have found all the same links with a Google search, Copilot undoubtedly made the process faster and less tedious. 

I don't generally need to check half a dozen possible sources to confirm a point I'm making in a blog post. One will do just fine. If Copilot or Gemini can do the winnowing for me and cut fifty or so Google first page results down to four or five, complete with a short precis, that's very helpful. Not game-changing or paradigm-shifting... just helpful.

Which leaves us a good distance ahead of where we were last time I did this. So, progress! Whether it will really make writing posts any quicker I very much doubt but I think that's probably down to my inability to know when to stop typing rather than any shortcomings in the software. Maybe my next experiment ought to be seeing how well Gemini does as an editor.

I probably need one of those more than I need a research assistant anyway.

Notes on AI used in this post.

In this case, we're only interested in the pictures, both of which were generated within Copilot itself. That required me to log in with my Microsoft account, which disturbingly required just a single click and no details. I may need to purge my cache. The images themselves were created by  DallE 3, to whom Copilot delegates art duties.

The header is supposed to be the three named Chatbots, Copilot, Gemini And ChatGPT, relaxing in a bar after work. The full, somewhat overwrought prompt, which I failed to record for posterity, asked the AI to imagine it was the illustrator of a 1970s comic book, something which wasn't remotely evident in the resulting image. I also changed the image to landscape and re-envisioned it as a "block print", one of the offered options. 

The second picture is the result of the prompt "Here is a line taken from an article about AI: "What a bunch of fence-sitters! And how insufferably polite!" Please provide a sutable illustration. It can be in any style but must be a drawing or painting, not a photograph." I then altered the result in the same fashion as the first.

Both images are unmitigatedly tedious and uninspired. They absolutely look like they were created by AI, not least because each of them has issues with feet going in the wrong direction, alhough I guess we can't conclusively prove which way an AI's feet ought to go... 

I wouldn't use either of these images other than to make a point about the quality of Copilot itself in the context of this particular post.