Is Generative AI the Future of Video Games?
Discover more from AI and Games
Is Generative AI the Future of Video Games?
We highlight the opportunities, and concerns, of the sector.
Artifacts is a new YouTube series made possible thanks to the AI and Games crowdfunding community on Patreon as well as right here with paid subscriptions on Substack.
Support the show to have your name in video credits, contribute to future episode topics, watch content in early access and receive exclusive supporters-only content.
Generative AI! It's everywhere! Every news site, every social media platform, people are talking about it. Using a suite of powerful new AI tools that can create text, voice, or graphics with a simple prompt, has opened up all sorts of exciting applications. You can use it to Bing, write sheet music, write the lyrics that go along with it, write code for vector graphics, and animations, and even plagiarise that college essay you've been putting off all semester (well not you lot, my audience are lovely people who respect academic integrity, don't go proving me wrong now).
However, amongst all of the big reveals, the elaborate press events, tech demos, and social media hype, it's difficult to truly understand what the state of the art is, and whether it is actually going to have an impact on the video games industry. Is generative AI ushering in a new age of tools that will make game development easier, affordable, and more accessible? Will AAA games now only take weeks, even days to make, instead of years? Are we going to get the source code and assets for games like Skyrim just by typing a prompt into ChatGPT? The answer, and the reality that goes alongside it, is much more nuanced than a soundbite bouncing around Twitter. At the risk of violating the creed of the internet, I'd like to dig a little deeper into what these technologies are, how they work, the issues that surround them, and their production readiness in the context of how video game development traditionally works.
For this entry of Artifacts, I tackle the big generative AI question that continues to haunt my news feeds and highlights that the path to generative AI becoming a useful tool in the games industry, will be a slow and gradual process. Plus, stick around for the end as I ask you for your help in truly understanding the impact this is all having on developers, content creators, players, and everyone else in between.
Hey, I’m curious: what are your thoughts on generative AI? After reading this article, please check out the reader survey I’ve created where people can share their thoughts on generative AI for games. If you’re a game developer, content creator, student or just someone who enjoys playing games, I want to hear from you. It should take you at most, 10 minutes of your time, and will be a huge help in contributing to future episodes of Artifacts.
Visit the link to the survey right here!
An Overview of Generative AI
So let's start with the basics, when we say Generative AI, what do we actually mean by this? It's a buzzword that has dominated our newsfeeds for much of 2022 and into 2023, but it's on me to explain it to you. So here goes...
Generative AI is the more streamlined and marketable term for any AI system, typically one built using machine learning, that can create an artefact of some sort as its primary function. GPT creates text, Stable Diffusion and DALL-E create images, GitHub Copilot writes code, and Speechify generates voice audio. You get the idea. In each instance, they're provided input, and from that prompt, it creates something designed to evoke it. Whereas we typically assume an AI system creates answers to problems, it's seldom creating an output that we as humans can use in practical and creative ways. It's called Generative AI because it generates interesting artefacts for the user.
The term Generative AI caught on courtesy of the work by Ian Goodfellow and his collaborators in 2014 on the use of Generative Adversarial Networks or GANs, that can be used for image generation. GANs sit alongside another form of machine learning known as VAEs or Variational Autoencoders that emerged from research by Diederik P. Kingma and Max Welling in 2013 and has since proven useful for image generation and natural language processing. GANs and VAEs have been two driving methodologies for what we now call generative AI. While their underlying training approaches differ, what they share is a process in which they seek to understand the underlying properties and traits of the data they analysed and store it in a compressed format: a concept known as the latent space. If the system can process the latent space sufficiently, then it can later seek to decode it in new and interesting ways that still reflect the properties found within the latent space. Hence they can create for example images that look a lot like images analysed when it was being trained.
We've already explored applications of GANs for games here on this channel. Episode 39 of AI and Games highlighted the use of GANs to generate Super Mario levels, meanwhile in our overview of texture upscaling in episode 61, I looked specifically at how this technology was used as part of the art pipeline for the Mass Effect Legendary Edition.
While these breakthroughs are impressive, it is important in the interest of historical accuracy, that I point out none of what we're seeing in these systems is conceptually new. This work is all built upon decades of existing research in machine learning, and computational creativity. Generative AI is fundamentally a more marketable term for procedural content generation, or more specifically procedural content generation using machine learning. The idea of systems using AI and machine learning to generate output for creative purposes is a well-worn one. And it's certainly a topic we've covered here on this channel, ranging from computationally creative systems like ANGELINA - the AI system that designed its own video games back in episode 20, all the way back to particle weapon generation in the shooter 'Galactic Arms Race' which was way back in episode 05. While the term ‘PCG-ML’ has in fact been adopted in some scientific research circles for some years now, it clearly doesn't roll off the tongue as effectively as 'Generative AI'.
However, the big reason generative AI is making such headway is the recent gains in deep learning that resulted in significant improvement over previous generations. The output of GPT4 is a significant improvement over the same system even a year ago. Meanwhile, the likes of DALL-E and Stable Diffusion are equally impressive and highlight the potential of this technology to a great degree. Plus thanks to cloud and edge computing infrastructures, it's now much more accessible for a regular person to utilise, with text-based interfaces on web pages or mobile apps. You type questions to ChatGPT for it to answer, and you tell Midjourney what type of images you want it to create. By creating such a simple interface, it helps not only make it more palatable for the average user, but also maintains the mystique of it all: that it reads your text, and then acts upon it, making it seem much more intelligent than it truly is.
The Changing Market
A big part of that mystique is the aforementioned gains in overall output and performance. The big application areas of generative AI - such as text generation, speech-to-text, text-to-speech, and text-to-image - all of which have been in some form of development for decades now, have seen significant gains courtesy of three main elements:
The improvements in Deep Learning AI, notably in the development and training of large-scale convolutional neural networks.
The access to large-scale datasets allows for a richer understanding of the underlying feature space, and the ability to produce a broader range of responses that better align with our expectations.
Lastly, there is simply the sheer process power that cloud and edge-based computing now afford. Training large-scale systems in the cloud, and then deploying them such that they're readily accessible on desktops, laptops, phones and any other devices that can quickly connect courtesy of an internet connection.
There is, of course, the fourth underlying element - money! In the past 10 years, we've seen a huge shift in how AI research and development is now funded. Since the inception of the field in the 1950s, the bulk of R&D in AI has been conducted in university research labs, with the majority of funding being provided courtesy of government research grants and perhaps with some larger businesses throwing in money either to support specific projects or research fields. The proportion of AI research coming out of corporate labs was relatively minimal. Nowadays, as is evident by the owners of many of these AI systems, it is corporations taking on the bulk of the work, and at a scale previously not plausible due to the sheer amount of money being thrown at it. This has led to a rat race of sorts, as companies ranging from the biggest corporations to the smallest of startups push hard and fast to make their big announcements. Investors are pumping billions into the sector with the prospect of massive returns. 2021 saw an estimated $70 billion dollars invested by venture capitalists, with around $46 billion in 2022. When you consider that other speculative areas such as Web3, NFTs, and VR are largely waning, plus recent headlines for the likes of GPT4, you can bet that investment in Generative AI is only going to continue.
All of this has led to big pushes for new platforms, embarrassing false starts, the odd retraction or rework, and a mixture of hype, and enthusiasm but also some understandable apathy and frustration with the state of generative AI. Given the underlying financial incentive, be it to be the first to market for customers, or simply to direct even more investors their way, we're seeing new AI tools, systems, features, tech demos, and the like being announced on social media or via press release on an almost daily basis. Capitalism is operating in high gear, and only time will tell which of these companies will still be standing when the dust settles.
Generative AI for Gaming
So having painted something of a broader picture, what value is all of this in the context of games?
There is a huge amount of potential in moving games forward through the use of generative AI techniques. Using AI to create textures and sprites, generate animations for specific actors, writing descriptions for quest logs or lore bibles. Generating storylines for role-playing games. Creating real-time conversations with non-player characters that are in-world, relevant and react to player input. All of these are attainable, and in many respects developers of all shapes and sizes can start using these tools today. An indie developer could use Midjourney to kickstart their ideation process, helping establish mood boards and concept art. Meanwhile, a programmer may generative programming tools such as GitHub Co-pilot to assist in writing code for a new feature. We're in a new age in which generative AI has the capacity to change how developers make games. It's a notion that everyone from the smallest of start-ups to the biggest of corporations such as Microsoft, Google, and NVidia has jumped on with great enthusiasm. With the recent announcements of Nvidia's ACE platform for NPC creation, through to the likes of Inworld AI's Origins demo being something you can wishlist on Steam, it's catching the attention not just of developers, but players as well. I mean geez if I had a penny for every comment on my Facade video saying 'hey imagine this but with ChatGPT' I'd... well I wouldn't be rich, but I’ve paid for my coffee today that’s for sure.
But regardless of their desire to corner this part of the market, the path towards success in this space is not a straight line. Building these tools such that they are safe to use, appropriate for developers’ needs, and will not result in the developers suffering public embarrassment or even facing litigation is an ongoing one. Yes, as I write this now, no doubt many people can point to an example on social media of someone proving it's possible and I'm just being a mood killer (that's Dr Mood Killer to you thank you very much). I mean to make the point for them, here's BenBonk using ChatGPT to generate code for a Unity project, or more recently SamYam making a game prototype in an hour using generative coding tools. But as always, the path between 'Oh hey we made this work for a TikTok' and 'this is a shippable commercial product' is vast, legally unclear, and contains a myriad of pitfalls along the way.
We'll get into some of the issues that reflect the current state of the art in a moment. But it's worth pointing out, that there are a myriad of companies out there trying to solve the problems I'm about to address. AI Dungeon creators Latitude alongside the likes of Hidden Door are working on language models for story generation in games. Ubisoft is experimenting with text generation for script writing. Inworld and Convai are exploring how to make more realistic avatars with workflows that plug into game engines. Meanwhile, Unity is exploring how to interface Generative AI into their workflows, while Roblox already has tools that are available for their creators to use. The state of generative AI for games is going to evolve drastically in 2023 and beyond, and my hope is that we'll be digging into these in future videos here on AI and Games. But right here, right now, these are some (not all) of the big issues impacting the field.
The Technology Issues
I mean first and foremost let's just start with the basic reality check: while we have seen orders of magnitude improvement in generative AI systems, the technology is often nowhere near as capable or powerful as it is often advertised. While there are many hype merchants advocating that Generative AI is going to transform the games industry today, we're still months if not years away from seeing many of these technologies be as reliable, sustainable, practical and seamless as the tech demos suggest. It doesn't help of course that this is right off the back of the Web3 and NFT hype, which also suggested it was going to transform the industry. And yeah, I suspect a lot of people changed their LinkedIn bios sometime in late 2022 as they changed the bandwagon.
But unlike NFTs, AI does have the potential to change game development in a significant way. But these technologies and their accompanying tools and integrations are all evolving at different speeds around the world as different companies take on the big challenges of how to make this tech more palatable, with some already having a head start and making real gains in the field. But, this is just the start of a longer journey as generative AI becomes more palatable for practical and commercial use.
It's a lot of work to go from a cool tech demo to an established toolchain ready for developers to use. And it's important that when a new demo is promoted, that the reality checks kick in. Sure a tool can generate a cool bit of dialogue, an interesting-looking texture, or even a complete non-player character or game level. But does it produce it at high quality 100% of the time, every time? If there are still issues and risks that require human intervention, then that inhibits their efficacy if they're advertised as solving problems without user involvement. Plus, it's worth highlighting that a lot of generative AI tools need to solve problems game developers are actually facing and in a way that game studios operate. I'll be coming back to this later in the video, but it's important to recognise that games, even from the smallest of indies to the biggest of AAA, are complex productions often with multiple people operating on the same projects in different capacities. What generative AI needs to be doing is solving problems that game developers actually face, and help bootstrap productivity, rather than seek to replace them.
In my recent AI 101 episode on the use of Machine Learning in the games industry, I highlighted that one of the big issues that prevented it from being used until recently was that the techniques being created and the problems that researchers were trying to solve, were seldom the things that the games industry saw as needing to be solved. It's only now with ML-powered techniques solving specific production challenges that has taken on a whole new relevance. I'd encourage you to watch that video if you haven't already given the arguments made for machine learning for games as a whole are highly relevant to the conversation we're having right here about Generative AI.
The Data and Legal Issue
Now moving into the practical elements, perhaps one of the biggest issues that surround a lot of generative AI tools, and this is not just in the games industry, but the field as a whole is the data used to train them. Text generators need a significant amount of text in order to learn from, build open and subsequently write their own. Similarly, an image generator needs access to a significant corpus of image data from which it can then generate as well. Of course, the issue then is where that data come from, and critically, the rights of the individuals who created that data.
The vast majority of generative AI tools are using training data that isn't open source, and in some cases, has not been outright declared what exactly is in it. This leads to two problems for a user of these systems. First that if you don't know what's in the training set, that leads to unknown outputs and an inherent lack of trust in these systems. After all generative AI systems are, in essence, very complex copycats. A text generator mimics the text it reads, and an image generator mimics the art that is in the training corpus. We want to know it's being trained on relevant data such that we not only get practical outputs but also avoid undesirable ones. A character spewing racist, sexist, homophobic or transphobic rhetoric, because it learned to say it thanks to a training set that incorporates the likes of message boards and chat forums, isn't going to prove palatable for a big-budget AAA release.
Now compounding the issues of data and access, are also the legal ramifications of these generative AI tools. To say that there are serious legal issues surrounding generative AI is, quite frankly, a generous understatement. The source of the aforementioned data is a big problem. Image generation tools have been found to contain within their datasets the work of artists who did not provide their consent, and research has found ways to be able to effectively pull complete images out of generators that are nearly identical to those in their training data, only helping highlight their culpability. The significant backlash from the online art communities on the likes of ArtStation has arisen to prevent companies from scraping art without consent, while also seeking a boycott of AI art being posted on the platform and outright legal action as artists demand their work be removed and they be summarily compensated. And of course that's just in the art world: there's the ongoing legal case that Microsoft, GitHub and OpenAI are facing on whether their code generation tool learned to program by effectively breaching licenses on copyright source code assets which has huge ramifications for the future of data acquisition for generative AI systems.
This of course brings us to the issues of copyright and fair use. If AI-generated art is used as part of a game, be it concept art, character designs, textures and more, can be found to have been derived from an artist who did not give consent, this has big ramifications not just for the creator of the generator, but any art that could be gleamed to have derived from it. No big studios are willing to tackle the legal ramifications of this, lest they can guarantee they've built their own internal training data. Something which many studios are now actively exploring.
However, it doesn't solve the other legal problem: claiming ownership of the generated assets. Currently, there is no legal basis for AI-generated artifacts to be given the protection of copyright status. Just because you told ChatGPT to write those lyrics doesn't mean you own them. Similarly, asking Stable Diffusion to create an image, you don't own the image. GitHub Copilot can write the code, but you don't own the copyright of the code. Nobody does in fact. It's simply an asset that anybody can use. This prevents a lot of these assets from being useful in a production whereby you're building a commercial product. Given you want to have copyright over all assets and have sought an appropriate license for the use of those you do not. If not, then you run the risk of people stealing and using those assets however you see fit. This is going to be an interesting development in the coming years as countries around the world seek to tackle the myriad of legal issues that impact AI systems.
The Source of Creativity
Now that we've examined what are arguably the big issues that immediately impact generative AI for games. There's what is, to me, the core problem that the vast majority of media hype conveniently avoids: humans, people, y'know, the people that make games, are the source of creativity in games, and generative AI needs to support that.
Much of the discussion around generative AI has focussed almost exclusively on using AI to effectively 'replace' human input. The aforementioned debacle surrounding generative art and the theft of assets has helped fuel the narrative that AI will effectively replace creatives. This is of course then fuelled by speculative hype suggesting generative AI could replace texture artists, concept artists, 3D modellers, animators, sound designers, voice actors, and even to some extent programmers as well. It's a classic case of conservative capitalism advocating for the automation of work that people not only rely on for their livelihood, and built entire careers around, but also the work they actually enjoy doing. And of course, many a company has jumped on this bandwagon, advocating to investors that they seek to yield increases in profit margins by simply laying off humans, and making the AI do all of their work.
Now in all fairness, there is a lot of work out there in every industry that everyone would benefit from being automated. And to be realistic about it, generative AI is going to change how many individual jobs and roles are performed, not just in games, but in virtually every business sector. But that's the key part: it changes how the job is done, and it will do that by supporting them to do their job. Now I'm not naive, some jobs will be heavily affected and some teams get downsized, but the situations where people advocate for AI to replace developers are downright foolhardy at best, and at worst, fucking stupid. Games are a creative medium, and we require developers to be at their creative best to get the very best games. It's that simple. Sure, AI can help support that and facilitate that, but critically the AI needs to be built to help creators achieve their creative goals, not seek to supplant them.
This is perhaps indicative of the often justified backlash against generative AI: it is posited as a means to replace someone working in a creative field, rather than a mechanism to support them to do their job better. This ultimately will not work, given you will always need humans in the loop as part of that creative process in order for it to work effectively. So, why not just get the AI to work as a tool that enables them to do the job more effectively?
Returning once again to my AI 101 on Machine Learning for games, I highlighted many examples of how ML is now being deployed in the industry. Whether it's texture upscaling, motion matching, cheat detection, or any of the other applications I mentioned in that video, they all have one thing in common: they're helping solve existing production challenges in ways that are faster and more effective than throwing more manpower behind it.
Do you want a simple example of this, and where it should really be going? Look no further than Speedtree: a technology used by pretty much every AAA game in existence since Elder Scrolls Oblivion to create trees in their games. Heck, it's even been used in dozens of major motion pictures, including several entries of the Marvel Cinematic Universe, since SpeedTree Cinema was released in 2009. Do you know why? Because artists don't want to have to sit and manually create tens of thousands of trees. And they also don't want to just give a video game level to a generator and ask it to ‘tree it up’.
But, if a tool like SpeedTree can help that level artist get the trees in there faster, at high quality and at scale, then they can work to make it an engaging and interesting play space. It's speeding up production by giving artists tools to help them do their job faster, and to a higher and more consistent standard. Until generative AI reaches that level of quality and interactivity, it's not going to prove all that effective.
Wrapping Up
With this article, I hoped to paint a broader picture of where we are with the Generative AI hype, given it's going to be an increasing focus both here on Artifacts and across AI and Games in the future. As discussed already, we're seeing a lot of these technologies begin to mature, and some companies are getting ready to showcase their products as useful tools for game developers to work with. In fact, at the time of writing, I am already preparing a handful of video projects in which we're going to see some of these in detail. But for now, it's worth highlighting what the state of the field is right now and providing a more grounded take. No, we're not in some Generative AI nirvana right now, nor will we ever be. But it's clear that there is great potential in a lot of these technologies in enabling game development to be more streamlined, higher quality and potentially more accessible.