Tuesday, August 20, 2024

In Which I Am De-Railed By AIs. Again.

Whether Wuthering Waves is as popular or more popular or less popular than Once Human is unknowable, since we don't have precise figures for either of them. If it was on Steam, at least we'd be able to make a direct comparison but it isn't, so we can't.

What we can do, though, is ask a friend. Or in this case, an AI. I asked Google's Gemini first, since Google seem very keen to position their little darling as a Personal/Research Assistant. Is it up to the job? Let's find out!

Gemini, as it tends to be these days, wasn't keen to commit. After a rough start, back when it was calling itself Bard and seemed happy to tell you whatever you wanted to hear, regardless of whether it was true, Gemini has turned into the cagiest, most cautious of counselors. I asked it

"Can you give me some population numbers for the video game "Wuthering Waves"? How many people play it? How popular is it?"

Gemini's reply was long on caveats and cautions but very short on hard numbers:

Gemini does like to editorialize. I didn't ask for a prognosis on the game's future health but I got one anyway. I also wasn't very impressed with the lack of hard data. I specifically asked for numbers but  other than the vague "over 100 regions" I didn't get any. 

I knew Kuro had released some PR about certain download or registration metrics they were pleased with, so where were those? I thought I'd get a second opinion so I asked ChatGPT.

I haven't been in the habit of asking ChatGPT these kinds of questions because for a long time it worked purely from stored data that had a cut-off point of several years ago. If you asked it anything about the present day it would warn you it had been trapped in time like some robotic Rip van Winkle and couldn't even tell you the name of the current President with any certainty.

That clearly wasn't going to fly in this brave new world of AI-at-your-fingertips so now ChatGPT does what I always hoped it would when you ask it something like this: it does a web search and collates the results for you. 

It pops up a little message telling you it's checking the web and it provides the links it used in the final result. That does inspire a lot more confidence in its findings than than all the fudging of Gemini's "it's safe to says" and "we can get an ideas".

I copy-pasted the exact same question and this is what I got back:


 Now, I have to say straight away that one of those references is from a site I have never had much confidence in - MMO Populations. If you want to know why, I submit their entry for New World as evidence. 

Since we have actual figures for New World via Steam, we know it's not doing so well. MMO Populations has that data, too. It even quotes it accurately: "New World is estimated to have 5,724 players per day this month." and yet it claims "New World is estimated to have 15,063,776 total players or subscribers." Despite that low player-count, the site still ranks New World #18 out of the 140 MMOs it tracks. 

Meanwhile, Wuthering Waves with over 90k daily players as shown on the same website, comes in four places below at #22. Clearly some disconnect there - and I won't even get into the bizarre list of games the site claims are "MMOs".

Discounting that one, of the other three sources, Esport.net is as vague as Gemini and anyway the post linked is from the first week the game went live. Playerauctions.com looks to stand on firmer ground and at least it explains how it gets its numbers:

 "Using our algorithm based on Google Trends, Wuthering Waves’s player count for August 19, 2024 is estimated to be 260,700 players, compared to the previous day, this represents a -0.23% decrease. In the last 7 days, Wuthering Waves’s player count peaked at 261,300 and reached its floor at 252,000."

Google Trends is reckoned to be a fairly reliable indicator, I believe, and those figures are at least bang up-to-date. You might have thought Gemini would have known about them, what with it being a Google service as well, but it often seems to be the case that one part of Google doesn't know what any of the others are doing.

ChatGPT's fourth source is somewhat redundant, being nothing more than a referral of my inquiry to Bing. I checked the results and they include the other three sources so I'd like to think ChatGPT picked them as the most accurate or reliable but who knows?

However the data was gathered, I'm reasonably comfortable, now I've back-tracked to check the sources, that it's as accurate as we're likely to get. I'm also quite content to call Wuthering Waves a popular and successful new game, which was what I believed but didn't just want to claim without corroborating evidence. Although whether it's an MMO, as some of these sources seem to believe, is another matter altogether.

As for using the AIs actually saving me any time over just running a Google search and clicking through to the most likely-looking results... yeah, that's not happening. I could easily have done the leg-work myself just as quickly.

Of course, if I'd been willing to take the AIs at their word, then it would have saved me the best part of an hour! It took both of them about five or ten seconds at most to come up with the goods. 

Unfortunately, we're still a good way away from from the day I can feel happy just asking the AIs a question and dropping whatever they come up with into the post I want to write. What still happens every time I try it is that instead of posting what I thought I was going to, I end up writing about the AIs and their processes instead. 

Oh! Look! It just happened again!

I guess my post about how really, surprisingly, very good indeed the recent update to Wuthering Waves was will just have to wait 'til next time. 

Look on the bright side, though. At least by then I might have played through the second story in the new, four part quest sequence. It'd be good to be able to check whether it keeps up the impressive standards of the first before I start going on about how amazing it is.

Probably all for the best then. Thanks AIs!

4 comments:

  1. LLM AIs do not reason, don't know how to reason, can't be trained to reason and the whole "reasoning" thing is not in their code. Some AI are going on the way of learning mathematical reasoning as an approach to physical reasoning... Which is tiny step in the right direction of AIs having some ability to understand a tiny sliver of the real world. But it's a long way ahead and meanwhile tbe current AI bubble is bursting as companies meet the onerous hardware costs of LLM AIs and face the prospect that nobody knows how to earn back those billions they're investing by selling... what exactly?

    ReplyDelete
    Replies
    1. They don't need to be able to reason to return accurate results from a search. They just need to do what ChatGPT did here - search accurately from the prompt and collate the results that most closely fit the parameters given. Bringing in all the cruft associated with the term "AI" from decades of popular culture just confuses things.

      If you're talking about the context in which I used the AIs in this post, we're really just looking at an improved search facility. They don't need to make anything up, just find it, sort it and summarise it. If there hadn't been all the hysteria about "AI" and these functions had just been described as "algorithms" or even "apps" we'd all just have got on and used them without thinking about it.

      The LLM/AI aspect in this usage really only needs to frame the results in natural English, something that software is designed to do and is very good at. The problem has been stopping them making stuff up but that really shouldn't be an issue if all they're being tasked with doing is looking at other websites and precising them. If the data is still wrong, then it would have been wrong on a basic Google search, which is why it's still down to the human who asked the question to cross-check the results.

      As you can see, when I did cross-check ChatGPT, it proved to have given me exactly the same information I would have found for myself before AIs were available. The problem at the moment is that, because of over-hyped claims made for them and the very inept way they were introduced last year, none of us trust the things yet. If they evolve into the dumb fetch-and-filter apps they always neded to be then they're goign to be very handy. I think we're getting closer to that now the ludicrous expectations have been proved false. Yes, it may bust the bubble, but the underlying usefuleness of the technology isn't going anywhere, any more than e-commerce vanished when the .com bubble burst.

      Delete
  2. It's funny, in on of the Star Wars prequals (maybe Clone Wars), Obi Wan remaks to Anakin, "Well if robots could think, there wouldn't be any of us." I always found that line thought provoking, becasue in that universe CPO30 and many other robitic character are clearly depicted as sentient. Did this mean that in the universe robots only seem to be sentient? They sure do react with seeming intelligence to a wide variety of stimuli, how does that even work?

    But now in a world of ChatGPT the line makes a lot of sense. It's suddenly easy to understand how something could behave as if it were sentient in many settings, and yet still not have anything close to "general intelligence" and thus the potential to trully understand any of the output it generates.

    Apologies for the off topic ramble. Good post!

    ReplyDelete
    Replies
    1. Not off-topic at all! I'd say that if C3PO can't think then we need to re-define what we mean by "thinking". As for our current AIs, though, the whole "Are they intelligent?" thing is a total red herring . Of course they aren't! The "Intelligence" in the name is about as literal as "Eternal" in Rome, The Eternal City. It's at best a metaphor but mostly just some efective PR.

      A much more interesting question in my opinion is can they do what the companies promoting them say they can? And the answer seems to be it depends. They're quite effective at some tasks, absolutely hopeless at others. I want them to do my research for me much faster than I can do it myself and sufficiently reliably that I don't need to do more than a general sense check when I edit. Last year they were absolutely nowhere near being able to do that. Now they are a lot closer. But still not close enough that I can trust them without close supervision.

      Maybe next year!

      Delete

Wider Two Column Modification courtesy of The Blogger Guide