How Inworld's Origins Demo Actually Works

After analysing its behaviour myself, I find out how it really operates...

This episode of AI and Games is sponsored by Inworld. Check out my assessment of the Origins demo which is out now on Steam, and then sign up with a free account, and start building your own AI characters using Inworld’s tools.

---

We're backing taking a look at the work happening at AI company Inworld. In a recent sponsored post, I detailed my time taking a look at Origins: the detective game created by Inworld and built in Unreal Engine, that acts as a tech demo of their conversational AI systems. Origins is arguably one of the first fully playable games that exploits the recent improvements in the likes of large language models, and voice pipelines for recognising and generating speech. While it's but a demo, it does highlight the potential this tech has for future game productions.

Today, we're digging a little deeper into how it all works. I had the pleasure of sitting down with Kylan Gibbs, one of the co-founders of Inworld and their CPO or Chief Product Officer. For this sponsored episode, I'm going to give you the highlights of our conversation and find out more not just about how Origins works under the hood, but what else we can expect to see from Inworld in the future.

---

AI and Games is a YouTube series made possible thanks to crowdfunding on Patreon as well as right here with paid subscriptions on Substack.

Support the show to have your name in video credits, contribute to future episode topics, watch content in early access and receive exclusive supporters-only content.

---

The Initial Response

One of the first things I wanted to ask about, was the overall reception to Origins since the release on Steam. What they’ve observed from watching users engage with it, and the the highlights for them having worked on it for so long without public involvement.

Kylan Gibbs [Inworld CPO]: Origins has been a super interesting project because we started the company in like Q3 2021. And around that time, like our first demos that we did actually were like these VR demos where I was spending 4 hours a night putting on this headset, interacting with these characters. And I can tell you they were not very good [laughs].

So Origins became sort of this internal testbed that we had to try and actually make this into a game because the problem is, especially before, is people didn't even really think that this was possible.

And so over the last year and a half or so, we developed it as a way to sort of demonstrate a lot of features that we have, to sort of actually inspire a lot of features that we were to develop. And doing that first-hand taught us a lot.

We finished Origins almost a year ago now, and we were pondering a while ago whether we should release it. Given it is really just a technical demo. It's not a full game. And, you know, gamers and game developers have high standards. However, on the release, it's been fantastic.

Establishing Narrative and Character Goals

Having discussed the response to Origins, I wanted to know a little more about how it all actually works. The Inworld tools have been evolving at a fair pace since the demo started development, and as stated, it's been finished for some time now. So getting a grasp of which features they utilised, and which version of them they had adopted, was something I was keen to find out.

So I started out specifically in the world knowledge and embeddings systems. As discussed in my last article, Origins runs a Character engine that allows designers to craft information about a specific character: their personality, backstory, the manner of their language. Now these characters are derived from the use of Large Language Models (LLMs), popularised by the likes of ChatGPT. The text provided by the designer in the character engine editor, creates ‘embeddings’ that that provide a supporting scaffolding of information that the language model use at a later date. In the context of Origins, that means things like a characters name, their personality, but also their motivations and goals they seek to achieve as part of the established fiction.

But how this all works, and the underlying infrastructure is becoming increasingly more complex than running a single language model.

Gibbs: I think where we where we've landed is like, you can't do this with a single LLM. We have I think 35 models that are all running in parallel now just to have a single sort of, you know, human like interaction, that's the ‘character engine’.

And then we have the ‘contextual mesh’ which is on top of that. And the idea behind that is, you know, as a, as a creator, you have this idea of a story world. You know, you have locations, you have certain scenes that you're within. You also have specific sort of reactions to how a player interacts. And then you as as a developer kind of set up that mesh, which is really a set of scenes, contextual knowledge, pieces of information [stored in a graph] that are all triggered depending on how the player interacts.

And then depending on how the player is moving through that graph, each of those characters get specific sort of prompts and information. But mainly actually there's a kind of a knowledge system that sits on top of it, which is kind of controlling those facts. And then on top of that, we have what we call like a ‘long-term memory system’, which is then taking each character's experience or interactions with you, kind of like summarizing or synthesizing those into kind of you can think about that is like the reflections that a human would have on their day to day experience and storing those for long term.

But while this handles the narrative elements of the scene overall, there is still a need for characters to have their own goals and aspirations. This is handled by a separate goals and actions layer stored inside the engine.

Gibbs: We have this whole goals and action system now, which we're kind of rebranding into like a ‘logic engine’, which actually allows you to control the specific contextualized responses that a character would have in any scenario.

Origins is actually using our older goals and action system, which is much more trigger based. So basically what that means is you go into the scene and Janus, the character who first introduces the setting to you, is is prompted to give you this background information about your role about the world. And then as you progress, then you have to go and talk to different robots and characters to kind of uncover parts of it.

And depending on which ones you've spoke to before or what you've spoken about, they'll be triggered to give you new information. We're actually doing sort of like an overhaul of it as well with this new system, which will mean you can kind of interact with any character in any order. But depending on sort of which one you have, they'll kind of nudge you in the right direction.

And then when you get to sort of Maya, who is kind of one of the, you know, the final character who's sort of this, I won't spoil for the player, but the character has all the information of all the decisions that you've made and will respond according to that.

And so this is kind of what we're trying to do over time is is use this system to increasingly allow that control of how the characters respond in each context while giving the player increasing agency. and that path that they go through like how does that then feed into that dialog generation on top of that.

Managing Player & Character Personalities

The proposed update to the goal system, sounds like it addresses an issue I found in Origins that you'd perhaps yo-yo between characters to try and figure out what was the piece of information you're missing, and how do you churn it out of them.

But on top of this there's the individual characters and their motivations. Inworld's tools state that we can express a personality for a character, and even give them some motivation. So I was curious how all of that intersects with the character engine and narrative mesh that is generated for a given storyline.

Gibbs: So you can kind of think about each character's kind of preset, what we call a brain. And so that brain is effectively the thing that decides how to respond to those moments and the context. It's very, very similar to how we operate as people, right?

You have friends who are the last person I'm going to, you know, come and ask me to help me. Another one may be the perfect person.

And so we really do sort of use that brain as the way to maintain the coherence of the character while allowing anyone to really interact with any scenario.

It also puts limits on sort of knowledge. So, you can kind of think about the general process of, you know, inputs being things from the world, being things from the user that are being said, seeing things from other characters that go into the character.

That ‘brain’ sort of filters and also responds to. So in terms of how it responds emotionally, how it responds to the terms of the relationship changes, and then all of that gets fed to like a series of large language models, both for verbal and non-verbal behavior, which then finally output basically what is said and what is enacted, and that gets sent back to the game engine.

And so this infrastructure, as Kylan described, was built to support this idea that players interact with the system and get an appropriate response based on the character. But an issue that sat in the back of my mind while playing Origins, was while these characters embrace their position within the fiction, I wondered whether my role was being recognised on their end. Given one of the biggest problems I had when playing the game, was getting any of the non-player characters to stop talking.

Gibbs: So we have a we have like a basic concept of what we call player profiles, which is kind of the information of who you are as the almost thing about yourself as a character interacting with the other characters.

I think it's actually one of things that Origins helped us learn a lot is, you know, players want to play through every different way. And in some cases I want to say, you know, please don't talk to me. I just want to kind of talk at you for a while taking this information, and then we'll come back and chat later.

It's not a scenario that necessarily is designed for it's completely possible within the, you know, the context of the game design there. I think that's that's kind of the edge cases that we've found. And so what it's also taught us is to kind of find these common other archetypes of players coming in.

And so we're working with a lot of our actual are our major enterprise partners as well, and we identify those edge cases and to say, okay, if this person really wants to try and get the latest news from this character, but you're in a Lord of the Rings world, probably not the right response.

How does the character, you know, gracefully bring you back to the world In your case? You know, if you're trying to get the characters to be quiet and you just want to go around and look at the world, also, we need to kind of design for that scenario.

Building Up at Scale

Perhaps for me what is going to be a big test for the likes of Inworld, is scaling up these systems to handle not just more complex interactions, but also creating these NPCs at scale. In Origins, these conversational AI's only chat with the player, and not other characters. So my first thought, is whether bot-to-bot conversations will soon be attainable? Given that’s often a useful way for games to convey information to players.

Gibbs: It's actually one of our biggest pushes right now. We do have internal demos that are working so we call this like a multi participant system, so you can think about a few different scenarios. So one is, you know, player comes up, you have two or three characters in a conversation and let's say you're in an Assassin's Creed style scenario, you listen over to their conversation, you understand not to pick up in information, really cool mechanism.

And then you also imagine, okay, then that player enters into that conversation. How do you sort of orchestrate that, you know, that four way conversation?

So that's that's actually a huge push because I think it unlocks a lot of really important mechanisms within games, it introduces games [the idea of] relationship as a mechanic.

Meanwhile if you've played origins, you'll notice that the demo really distinguishes between the 'primary' characters, which clearly have had a lot of time and effort put into them given their rich backstories and ideologies. But then how does Inworld plan to scale that up so that all the other NPCs you see wandering around are sufficiently interesting to engage with as well?

Yeah. So the way that we're approaching this is like we're calling character templates and you know, the basic idea here is you imagine you created, you had a huge RPG world, you had, let's say, 100 different towns. Each town you wanted to have like, you know, three barkeeps, ten blacksmiths, five knights, whatever it was. And you wanted them each have slightly different relationships, but you also want them to be aware of all the other ones. And within the same world.

You basically can build a character template, and then what we have is a way to contextualize things. We have a concept of ‘common knowledge’, which is, okay, you know, what is the location that I'm in? You know, what is the general rules of the world that I'm in? Who are the other characters within that world? And then, for example, for different groups, like let's say you talk to an orc versus a fairy versus an elf, they may all then have like a different set of information about the specific species that they're that they're a part of.

And so all of those are different parts of information that are then fed to the character. And then for each of those, we can actually auto generate character descriptions as well. And so we're you know, we're working on a way to instantiate a world with 100 characters with these different parameters. You know, with roughly the snap of a finger. You know, they're all there and they're all alive.

I think that the way gaming in general is going towards more open world games, I think that will be really important.

User Safety & Quality Assurance

Now while I was largely on my best behaviour when I played Origins, the elephant in the room during any of these playtests is the notion of safety. How do you approach quality assurance for a game in which you do not have 100% control of what an AI is going to say? But that's just the beginning of an even bigger challenge, in that while we need to ensure safety, it shouldn't inherently limit the type of language being used by these characters such that more evil character feels less threatening. Fortunately Inworld have baked this into much of their product offerings.

Gibbs: So we have a very strong base safety system that controls, first of all, what the whether [the AI] can respond. So if you say something really incendiary, hateful, you know, the character will still in their sort of character be like, “you know, this is actually not something I really want to talk about. You know, what about this?”

And the way that system is built is as it comes in, we filter whether the player said something that is particularly inappropriate if not, okay, let's let it go to the model. Let's see what the character says. Does the character say something that is particularly bad in response? If so, we'll kind of regenerate with a, you know, an intentional approach, and then you kind of get your response out.

And there's a series of six layers of safety within that so that, you know, we need to be able to design our systems that if we're making a My Little Pony experience, that it'd be appropriate for children, right? If you're building experience for GTA [Grand Theft Auto], on the other hand, you as the creator would be very upset if you had My Little Pony Level safety [laughs].

And then on top of that, there's things like the kind of fourth wall system which allows you to kind of contain things within the specific game world. There's other things that we can do, for example, around like biasing towards certain topics and bias and character responses depending on the scenario that allow you to kind of control for those different settings.

Our kind of thought is there is like we don't want to be specific, too opinionated as to what creators build, but there are of course, sort of limits that we place on that just due to, you know, ethical boundaries.

Now, there's of course, challenges because if you're playing a medieval world or some medieval fantasy world and there's orcs in there, everybody may hate orcs. Then you get into some challenging questions, given that’s resulting in hate speech towards specific groups. We do though then at that point offload that responsibility on to the creator who then decides whether or not that's an appropriate thing to to include in that world.

And we do monitor logs. So there there have been points where we had to reach out to creators and talk to them and say, hey, you know, just by the way, we'd ask you to like, change these parts of the character. We are finding that it's violating our terms and conditions

And so with my time with Kylan coming to a close, my thoughts were on what's still to come? After all Origins was finished for many months prior to its release. Given the rate at which this technology is improving, what's in the pipeline from the company and does that mean improvements to Origins itself? Or entirely new projects?

Gibbs: We're starting to think a lot more seriously about first party experiences that demonstrate [new types of] games. There's like new new mechanics, new types of game designs that we want to inspire and have considered as well through conversations. So what I think we're going to start doing a bit more over the next little while is starting to think about first party content more seriously, not from the idea of tech demos necessarily, but actually full fledged experiences and games

So we are internally considering whether or not that means, you know, taking Origins or something like that, expanding that or trying to build a series of other experiences. But that is definitely something we're looking at and are also very keen for input on.

The greatest [third-party] groups that we've encountered have been like, we're not trying to take, you know, Legend of Zelda and make Navi talk. We're trying to take the Legend of Zelda and turn the characters into the Water Temple; where your actual interactions with characters is part of the game mechanism,.

Demonstrating that spark, though, may mean producing some of our own first party content that really illustrates what's possible.

Closing

And so it seems like there's plenty more to come from Inworld in the future, whether that means more of the Origins demo, or entirely new projects, well... we'll just have to wait and see. Naturally you can be I'll be keeping my eye on future events as they unfold. But in the meantime thanks for watching this episode, and a special thanks to Kylan for taking the time to chat with me about all things Inworld, and I hope you've found this interesting.