Best Paper: What's the Game, then? Opportunities and Challenges for Runtime Behavior Generation
Nicholas Jennings, Han Wang, Isabel Li, James Smith, and Bjoern Hartmann. 2024. What's the Game, then? Opportunities and Challenges for Runtime Behavior Generation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (UIST '24). Association for Computing Machinery, New York, NY, USA, Article 106, 1–13. https://doi.org/10.1145/3654777.3676358
Procedural content generation (PCG), the process of algorithmically creating game components instead of manually, has been a common tool of game development for decades. Recent advances in large language models (LLMs) enable the generation of game behaviors based on player input at runtime. Such code generation brings with it the possibility of entirely new gameplay interactions that may be difficult to integrate with typical game development workflows. We explore these implications through GROMIT, a novel LLM-based runtime behavior generation system for Unity. When triggered by a player action, GROMIT generates a relevant behavior which is compiled without developer intervention and incorporated into the game. We create three demonstration scenarios with GROMIT to investigate how such a technology might be used in game development. In a system evaluation we find that our implementation is able to produce behaviors that result in significant downstream impacts to gameplay. We then conduct an interview study with n=13 game developers using GROMIT as a probe to elicit their current opinion on runtime behavior generation tools, and enumerate the specific themes curtailing the wider use of such tools. We find that the main themes of concern are quality considerations, community expectations, and fit with developer workflows, and that several of the subthemes are unique to runtime behavior generation specifically. We outline a future work agenda to address these concerns, including the need for additional guardrail systems for behavior generation.
Transcript 2,759 words
Welcome. We're delving into runtime behavior generation in video games today, or RBG for short. It's quite the mouthful. It is. But hopefully, by the end of this deep dive, it'll all make sense. We've got some really cool research to unpack, so buckle up. It's an exciting area of game development. Games that could design themselves think about the possibilities. Games that design themselves. It sounds like science fiction, but that's essentially what we're talking about here.
It's less about replacing designers and more about giving them incredibly powerful new tools. Imagine a game where the possibilities are truly limitless, where the experience is different every time you play. We've seen procedural generation before in games, Minecraft's worlds, for example, or the randomized levels in a roguelike. Right, but that's often about the visuals, the layout, the look of the environment. What's different now is the potential to apply that same generative power to the way the game actually plays.
To the game's behavior, its rules, its mechanics. Exactly. And that's significantly more complex. Coding a system to generate a new room layout is one thing, but programming a system to generate compelling and functional gameplay rules, that's a far greater challenge. And that's where LLMs come in. Precisely. Large language models, the same technology behind those chatbots everyone's talking about. They can be used to generate game code on the fly.
So we're talking about AI that can write code, but specifically code that dictates how a game behaves. In a nutshell, yes. The research paper, What's the Game Then?, introduces Gromit, a research project exploring this very concept, using LLMs to craft unique gameplay experiences in real time. Gromit. Bitchy name. So how does it work? Let me illustrate with a scenario. Imagine you're playing a game, an adventure game, let's say. You come across a dusty old bookshelf.
Classic adventure game troupe. There's got to be something hidden there. Perhaps. Now in a traditional game, the outcome of interacting with that bookshelf is predetermined by the developers. They've coded specific events or responses. But with Gromit, things become far more dynamic. Okay. I'm intrigued. What happens when I try to interact with this Gromit-enhanced bookshelf? Gromit uses something called a semantic scene graph, or SSG, to understand the game world.
It's essentially the game's self-awareness, representing all the objects, their relationships, and properties. So the game knows what's what and how everything is connected. Yes. And crucially, it can use that information to make decisions on the fly. When you interact with that bookshelf, Gromit takes that action, combines it with the information from the SSG, and generates a prompt for the LLM. A prompt.
A structured way to convey the game's current context to the LLM, what's happening, what objects are involved, and essentially asks, what should happen next? The LLM then generates code in response to that prompt. It's giving the LLM a behind-the-scenes look at the game and asking it to improvise. In a way, yes. That code is then fed back into the game, compiled, and if everything works as intended, a new behavior is born.
All of that happening in real time while I'm playing the game. That's incredible. But what if the code the LLM generates doesn't work? What if it's buggy? That's a crucial consideration. And the researchers behind Gromit have built-in error handling. If the code fails, the system automatically tries again, tweaking the prompt or the LLM's output until it succeeds. So it's like having a tireless AI coding assistant working behind the scenes to make sure everything runs smoothly.
You could say that. It's still early days, of course. But the potential here is immense. So they've essentially created a system that can dynamically generate new game behaviors in response to player actions. Have they actually put this to the test, though? How do you even begin to demonstrate something like that? They create several demos showcasing Gromit's capabilities. A simple sandbox environment served as their initial testing around.
A sandbox. A virtual space where they could spawn objects and then use Gromit to manipulate them using the LLM. Players could give commands things like, make the apples spin. And it worked. With a surprising degree of success. Within that controlled environment, a simple sandbox designed specifically for these tests, Gromit boasted an impressive 85% success rate. 85%. That's incredibly high, considering we're talking about AI generating code to make things happen in a virtual world.
But I have to imagine it wasn't always smooth sailing. What happened when they took Gromit out of the sandbox and into a more complex setting? You're right to suspect some challenges. They integrated Gromit into a pre-existing traffic simulation, a much more complex environment, and the success rate dropped to around 30%. 30%. 30%. That's a significant drop. Still, not entirely unexpected, I suppose.
It highlights the complexity of what they're trying to achieve. It certainly does. And it underscores the importance of designing games with RBG in mind from the outset. If the game world is built with the understanding that the AI might try to do unpredictable things, you can mitigate some of those issues. Fascinating. So they had their sandbox demo where they could directly interact with Gromit, making apples spin and whatnot.
What about something a little closer to an actual game? Their next experiment involved an escape room, a perfect scenario for exploring how Gromit could generate game behavior within a more structured environment. Okay, I'm all ears. How does an AI-powered escape room work? They designed a small escape room with a relatively straightforward puzzle. Find a hidden key. Unlock the door. However, they made almost every object in the room interactable.
And behind the scenes, Gromit was ready to respond to the player's actions. So I could try to use a book on a flickering light bulb or a chair on a grandfather clock, that sort of thing. And Gromit would try to make something happen. Exactly. It wouldn't always be successful. Remember, they still had those guardrails in place to prevent the AI from breaking the game.
But the potential was there for the game to react in novel ways. And in many cases, Gromit did generate new functional interactions. Did they have any examples of players discovering solutions that the developers hadn't even planned for? They did. One particularly memorable instance involved a player using a torch on a bookshelf. Seems harmless enough. What happened? Gromit decided to generate code that made the bookshelf burst into flames, revealing a hidden passage behind it.
A solution that the developers had never even considered. Now that's what I'm talking about. That's emergent gameplay right there. The game itself is a co-creator in the experience. It really makes you wonder what other hidden depths are lurking in all the games we play. All the seemingly mundane objects that could be so much more if only the game knew how to react to our crazy ideas.
It's an intriguing thought, isn't it? It challenges our assumptions about what a game can be. It certainly does. But let's not get ahead of ourselves. They also had a third demo, right? One that focused on a more specific application of RBG. They did. This one was inspired by games like The Legend of Zelda, specifically. The concept of combining different spells or items to create new effects.
Ah, the classic. What happens if I mix this potion with this bomb scenario? Always a recipe for chaos. Precisely. In this demo, they focused on the spell combining mechanic. They had a set of basic spells already programmed, but Drawn IT was responsible for generating the effects when players tried to combine them in new ways. So instead of having a fixed number of spell combinations, each with a predetermined outcome, they had a system where the possibilities were practically endless.
That's ambitious. How did that work out in practice? It was remarkably effective. Imagine you have a fire spell and a wind spell. Pretty standard fantasy RPG fare. Indeed. Indeed. Now, in a traditional game, the effect of combining those two spells would be determined by the developers. But with Gromit, you're essentially asking the AI what happens when you combine fire and wind. It has to analyze those concepts and generate code that produces a plausible and hopefully impressive outcome.
So instead of getting a generic fireball plus a gentle breeze effect, I could end up with a raging firestorm that engulfs my enemies or a concentrated vortex of flame that sends them flying through the air. Exactly. It opens up a world of possibilities limited only by the AI's ability to translate those concepts into code. And it's not just limited to spells, of course.
Right. You could apply this to crafting systems, dialogue trees, even enemy AI. Imagine a game where the enemies learn from your tactics and evolve their strategies in response. Or a crafting system where you can experiment with combining any materials you find to create unique items with emergent properties. The possibilities are really quite staggering. It's almost overwhelming to think about. Overwhelming, perhaps, but also incredibly exciting.
It represents a paradigm shift in game development, a move away from the limitations of pre-scripted experiences towards something far more dynamic and unpredictable. But it's not all about the technology, is it? At the end of the day, games are created by people for people. That's a crucial point. And the developers of Gromit recognize this. After their initial demos, they conducted interviews with 13 game developers to get their perspectives on this new frontier of RBG.
Smart move. After all, these are the people who would be on the front lines, so to speak, integrating this technology into their workflows and grappling with its implications. What was their overall impression? Were they ready to hand over the reins to AI overlords? So not quite ready to embrace the AI overlords then. What did these developers have to say about Gromit and this whole idea of runtime behavior generation?
It was a mixed bag, to be honest. Yes, there is definitely a lot of excitement, especially about the creative potential. Many of them were already using AI tools, ChatGPT, for example, to help them brainstorm or prototype ideas. So the idea of having an AI that could actually generate functional code, that was really appealing. It's like having an extra team member. But this team member is an AI that can code.
Exactly. And one that can keep up with their creative process. They envisioned a future where they could work collaboratively with an AI, bouncing ideas back and forth. And the AI can not only understand their vision, but actually help bring it to life in the game. Sounds almost too good to be true. Were there any concerns? I can't imagine it was all sunshine and roses.
Oh, they definitely had concerns. Game development is a very intricate process. And these developers have spent years honing their craft. So the idea of relinquishing even a little bit of control to an AI, even a very advanced one, was understandably a bit unnerving. Rightfully so. So what were their main concerns? Was it about the AI going off the rails and creating something completely unplayable?
That was part of it, for sure. The possibility of unpredictable AI output is always a concern. But it went deeper than that. They were also thinking about the practical implications, how do you integrate this kind of technology into existing workflows? How will players react to content that's been generated by an AI? And perhaps most importantly, can AI-generated content ever really be as good as something crafted by a human?
Big questions. It sounds like they weren't just thinking about the immediate future, but also the long-term implications of this technology. Absolutely. They identified three main areas of concern. Developer workflow, community expectations, and quality. Let's take those one at a time. First up, developer workflow. How do you even begin to integrate something as potentially disruptive as RBG into the already complex process of game creation?
That was a big topic of discussion. Some of them saw RBG as something that could be used to generate entire games from scratch. Others were more interested in using it to enhance specific mechanics or to fill in the gaps between handcrafted content. So a whole spectrum of possibilities, from subtle enhancements to a complete paradigm shift in how games are made. Right. And that naturally led to discussions about control, how much control are developers willing to give up to the AI?
Because even with the best AI, there's always a chance of unintended consequences. I imagine so. You don't want your AI assistant accidentally turning your gritty war drama into a wacky kart racer because it misunderstood a line of code. Exactly. That's why the concept of guardrails came up a lot. Ways to limit what the AI is allowed to do. For example, you might restrict the AI to generating content for a specific area of the game, or even just for certain types of objects.
Or you could provide it with stylistic guidelines or thematic constraints to make sure it stays on track. So instead of just letting the AI loose on their game world, they'd give it a set of rules to follow, a sandbox to play in. Precisely. It's all about finding the right balance between creative freedom and maintaining the developer's artistic vision. But of course, developer control is only one part of the equation.
They also have to think about the players. Right, because ultimately the players decide whether a game is good or not. How do they think players would react to this whole RBG thing? Would they embrace the unpredictability? Or would they prefer the familiarity of a human-designed experience? That was a point of contention. Some of them felt that players would love the idea of a game that was truly unique every time they played it.
Others worried that players might feel like they had less agency, less understanding of the rules, if they knew an AI was pulling the strings behind the scenes. It's that classic fear of the unknown, especially when it comes to technology. We like to know what to expect. Exactly. And in a medium like video games, where understanding the systems and mechanics is so important, the black box nature of AI can be a bit daunting.
It raises some really interesting questions about agency and authorship in a game. If the game can design itself, to some extent, what does that mean for the role of the human developers? It's a whole philosophical can of worms. Yeah. But let's move on to what's arguably the most important factor, quality. Because it doesn't matter how innovative or groundbreaking your game is if it's not fun to play.
Absolutely. And that was a major concern for the developers. While they were impressed by Gromit and what it could do, they also stressed the importance of making sure that AI-generated content is held to the same high standards as human-created content. Which is no easy feat, considering how polished and sophisticated modern games have become. Exactly. It's not enough for the AI to just generate something functional.
It has to be engaging, well-paced, and visually consistent with the rest of the game world. It's one thing for an AI to generate a sword. It's another thing entirely for that sword to feel good, to use in combat, and to look like it belongs in the game world. A lot of challenges lie ahead, but the potential rewards are immense. As AI technology continues to evolve and as developers get better at guiding and shaping AI creativity, I think we're going to see some truly incredible things happen in the world of video games.
And who knows, maybe one day we'll look back on all of this and wonder how we ever lived without games that could design themselves. Thanks for joining me on this deep dive into the world of runtime behavior generation, or RBG, and the incredible potential of AI in game development. Until next time. A pleasure as always. A pleasure. A pleasure. A pleasure. A pleasure.
A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure. A pleasure.