Harker’s Escape Lesson 3: LLM Analysis - Understanding Intent > Response Accuracy
A Year of Discoveries at Atomic Dirt
Until quite recently, interacting with a virtual assistant was routinely…underwhelming.
Siri a few years ago…
But since the dawn of LLMs, AI has made significant strides to understand a user’s intentions.
Gemini in 2024…
My sincere apologies to “Weird Al” Yankovic but clearly AI still has limitations, particularly in maintaining long-term coherence and logic.
If you jump on ChatGPT right now and input anything at all, I’ll give you much higher odds the LLM will understand what you’re intending to do vs. responding with an accurate response.
Current AI Strength: Understanding intent
The advent of LLMs and improvements in NLP (natural language processing) resulted in far superior “intent recognition.” Does this mean we can make a game that relies exclusively on text or voice commands? Maybe!
In Harker’s Escape, you could type all sorts of commands, and our tech was skilled at interpreting what you wanted to do and relaying that to Unity.
Ex. you want to break the leg off of a chair? Any of these commands will get the job done:
Remove a leg from the chair with force.
Detach a chair leg using brute strength.
Break one leg off of the chair.
Tear off one of the chair's legs.
Wrench a leg from the chair.
Detach a chair leg by force.
Rip the leg clean off the chair.
Pry a leg off the chair.
Forcefully remove one leg from the chair.
Yank a leg off of that chair.
Disassemble the chair by removing one leg.
Extract a leg from the chair.
I’m not sure why you’d say “extract a leg from the chair” but hey, it’s America!
A more flexible understanding of player intent is wonderful progress because it unlocks a world of more accessible gaming and potential to create new user experiences.
Unfortunately, the road to unplayable games is paved with good intentions.
Current AI Weakness: Long-term coherence and logic.
After a harrowing experience at Texas Road House, I used Google Gemini to learn more about how the USDA grades steak…
An incredibly creative name (it’s pronounced “meat·muhn”) for an incredibly personable butcher.
And soon enough, I learned the term “no roll” cuts…
And a few interactions later…
Now, is this the best example of coherence drift? Probably not. But maybe you learned something about meat from Mr. Tom Meatman!? *Record scratch* Sorry, you learned nothing. No roll beef is beef that is sold ungraded, so Tom Meatman is talking nonsense. Meatmaaaaaannnnn!!!!!
Believable and engaging game worlds require consistent, coherent environments and narratives. There are myriad examples where over time the LLM starts to lose the thread. While most individual examples of coherence drift seem rather harmless, when repeated and compounded the world your game is building unravels entirely.
Takeaway: Meaningful improvements in intent recognition are promising for accessibility. But AI, on its own, cannot be relied upon to handle the intricate world-building and narrative management required in a game.
And a few more examples of improved intent recognition for your pleasure…
Next on The Dirt: input method battle royale