GenAI in Gaming Industry Report: Q3 2024

Welcome to the first edition of the AI in Gaming Industry Report. Following our last article about "The AI Revolution in Gaming," we've decided to provide quarterly updates to help you stay informed on the rapidly evolving landscape of AI in games.

In this report, we'll dive into the latest developments and news, including:

• The most recent announcements and breakthroughs.
• Notable fundraising rounds and investments in AI gaming startups.
• Summaries of cutting-edge academic papers and industry studies.

For this quarter’s feature story, we look at the emerging trend of AI co-pilot for developers. From GitHub Copilot to Anysphere's Cursor, these AI-powered assistants are changing how games are coded.

Whatever your role—a game developer, an industry executive, an investor, or a casual observer eager to get a handle on what the future of gaming might look like in the context of AI—this report is designed to serve as your quick and up-to-date briefing.

In The News

• Electronic Arts, a widely criticized game studio and publisher, fully supported generative AI during its Investor Day 2024 presentation. CEO Andrew Wilson emphasized that AI is more than a buzzword; the company has over 100 active AI projects. The company also introduced its new Imagination to Creation tool, which uses AI prompts to create a video game in real-time.

• During an investor Q&A session, Nintendo's president, Furukawa, highlighted intellectual property concerns and the company's emphasis on unique experiences. While Nintendo is open to technological advances, it remains cautious, prioritizing dependable solutions over rushing to embrace new trends.
• After CEO Christoph Hartmann's comments about AI potentially streamlining game development, Amazon Games faced backlash. The company clarified that voice actors are still "essential contributors" and emphasized responsible AI use.

While big companies made headlines, the sector saw significant investment activity, indicating a growing belief in AI's ability to change how games are created and experienced. Let's take a look at the most notable funding rounds.

Investments

All of these startups raised $415.1 million this quarter, with World Labs ($230 million), Volley ($55 million), and Qodo ($40 million) leading the way. Investments support companies developing AI-driven game creation tools, social gaming platforms, narrative generation systems, and AI-enhanced game distribution channels.

The Latest Research

1. The GameNGen Doom project, a groundbreaking AI engine that allows neural networks to simulate playable video games, was probably the most talked-about story this quarter.

The AI imitates games like Doom with such accuracy that it is nearly impossible to distinguish between authentic gameplay and simulation, thanks to diffusion-based models similar to those used in image generation. However, the project still faces several challenges, including high computational requirements and the need for large amounts of training data. Despite these challenges, it demonstrates AI's growing ability to understand and reproduce complex game mechanics and visuals.

2. Building on the theme of AI comprehension in gaming, researchers presented VIDEOGAMEBUNNY, a specialized AI model for understanding game content.

Trained on a diverse dataset of over 185,000 images from 413 games, this model outperforms larger, general-purpose AIs in game-specific tasks such as identifying game elements, understanding mechanics, and spotting glitches.

3. To bridge the gap between AI understanding and player interaction, Rao et al. studied the dynamics of human players collaborating with AI-powered non-player characters (NPCs) in a custom Minecraft game.

Using GPT-4 for natural interactions, the study found promising potential for engaging gameplay and mutual assistance, which is especially beneficial to new players. However, it also revealed significant limitations in AI's understanding of 3D environments and real-time game states, resulting in inconsistent NPC behavior and spatial awareness.

4. Moving from understanding to creation, Zhang et al. presented a text-to-game engine capable of creating playable role-playing games (RPGs) from simple text descriptions. The "Zagii" engine uses advanced AI models to create game stories, characters, and visuals in real time.

With 803 games generated and over 60,000 gameplay sessions logged, this technology demonstrates its ability to democratize game production, making it more accessible and efficient.

5. Complementing the text-to-game approach, the Sketch2Scene system represents another leap in AI-assisted game development.

This unique application turns simple doodles into realistic, interactive 3D game worlds. Sketch2Scene's ability to automate the production of terrain, objects, and textures has the potential to streamline game world creation.

6. Researchers developed an AI system to evaluate and improve video game tutorials, addressing the crucial aspect of player onboarding.

7. The technology uses Vision-Language Models (VLMs) to evaluate tutorial screenshots and identify possibly confusing or unclear instructions. Tested on a tower defense game, this method promises to improve tutorial quality while eliminating the need for intensive human playtesting.

While this research demonstrates AI's potential for game comprehension and production, another key trend is emerging: AI-powered tools for game makers. Our feature story explores this trend, focusing on the rise of AI co-pilots in game creation.

The Rise of AI Co-Pilots

When we think of AI in games, we tend to think of improving the gameplay – more intelligent enemies, more realistic environments, or personalized content. However, another ‘wave’ is gaining momentum and is focused on solving problems present in software development itself.

Stack Overflow’s data from the 2022 developer survey reveals that 63% of the respondents spend more than 30 minutes daily searching for answers or solutions to such problems. This indicates that one would have spent 333 to 651 hours per week searching for answers on behalf of a team of 50. This inefficiency reduces the output and small-scales the overall project performance to the extent of compromising developers’ morale and productivity.

AI co-pilots are built for these inefficiencies and are designed to alter how developers code. A significant milestone was reached in June 2021 when GitHub launched a preview of its AI-powered programming aid, GitHub Copilot. This tool, which uses AI to suggest code when a developer starts typing, quickly became popular. Microsoft's latest quarterly earnings report reveals GitHub Copilot's remarkable growth: 1.3 million paid accounts (up 30% quarter-over-quarter), adoption by 77,000 companies, and over 1.5 billion lines of code generated daily. Currently, Copilot produces more revenue than GitHub had in all of 2018, adding $2bn in annual revenue run rate to GitHub’s revenue and 40% of the service’s growth.

Research shows that AI services that enhance developers’ efficiency could grow the global economy by $1.5 trillion in 2030.

The Stanford AI Index Report emphasizes the influence on developer productivity and happiness:

• 88% of engineers report enhanced productivity
• 74% find themselves engaged in more rewarding work
• 88% noticed a marked increase in task completion speed
• 8% higher overall success rate in task completion
• Tasks are completed 56% faster (reduction from 161 minutes to 71 minutes)

AI co-pilots increase efficiency and improve job happiness, allowing engineers to focus on more challenging and gratifying areas of their work.

Investors and founders have noted all of these benefits. As depicted in the graphic below, the market for AI-powered development tools is quickly developing, with considerable investment going into both startups and established organizations.
‍

With an overall funding of $2.73 billion across multiple ventures, it's evident that the race to produce the most effective AI co-pilot is on. Poolside, a prime example of this trend, recently announced a staggering $500 million fundraising to accelerate its progress toward artificial general intelligence (AGI). Poolside believes that software development will be the first field where AI outperforms humans, and they've created a new approach called Reinforcement Learning from Code Execution Feedback (RLCEF) to help AI improve its coding abilities. They aim to democratize software development, enabling anyone to create complex software without coding knowledge. Let's look at the basic capabilities of these co-pilots and how they might be deployed.

How does it work

Co-pilots for coding leverage machine learning models trained on large code repositories to generate real-time contextual suggestions. As programmers type, the AI analyzes the codebase and provides code snippets to help enhance and extend the written code.

Let's take Cursor, for example. It is a fork of Visual Studio Code that adds advanced AI capabilities to a familiar interface for most developers.

As you can see, Cursor can produce code on demand and even build an application’s entire framework. Developers can communicate with the AI in plain English about their code. Another fascinating VSCode fork is the Haystack Editor, a 2D digital whiteboard-based IDE that dynamically generates code connections as you navigate and edit files.

Game Engine Integrations

Integrating AI-powered editors into game engines is gaining momentum. For Unity, Prithvi Bisht's Medium article provides a step-by-step guide on how to use Cursor AI with Unity. Unity Code Assist Lite, a free Visual Studio plugin with over 50,000 installations, provides improved code completion, in-line visuals, and AI-powered support.

Moreover, there's also an in-engine plugin available on the Unity Asset Store that allows developers to control the Unity Editor using AI commands directly within the engine. This tool enables you to set up scenes, configure GameObjects, adjust project settings, generate scripts, and create textures simply by typing tasks. With features like full undo operations and detailed execution history, it significantly enhances workflow efficiency and flexibility for Unity projects.

Meanwhile, Unreal Engine developers are looking into tools like Bluepy, a plugin that uses OpenAI's API to produce Blueprint nodes from natural language inputs. Bluepy allows developers to describe the required functionality directly within the Blueprint graph, and the AI constructs the necessary nodes and connections.

But this is only the tip of the iceberg. Game creation involves unique challenges that differ from regular software development. These include complex physics, custom pathfinding, day-night cycles, and multiplayer networking. Large codebases can be intimidating for new team members, and tight deadlines necessitate quick onboarding and delivery while remaining consistent with the existing design.

Startups like Code Maestro are developing solutions beyond traditional co-pilots. These tools offer customized assistance for game-specific tasks, from AI behavior design to multiplayer debugging, while creating interactive knowledge bases to streamline onboarding.

Code Maestro's capability includes legacy code and custom engines, and local hosting provides data security, which is critical for protecting intellectual property.

However, the impact of these AI assistants on game production remains to be seen. We'll be keeping a careful eye on how these technologies fit into existing workflows and whether they can live up to the promise of more efficient development processes and creative gaming experiences.

Current Limitations and Future Potential

• GitHub Copilot's output often requires extensive testing and verification. A study found that 40% of AI-generated code contained vulnerabilities in high-risk scenarios.
• Developers like Darren Horrocks worry that relying too much on AI can lead to a misunderstanding of core ideas and encourage lazy development methods.
• Intellectual property and ethical considerations include the possibility of disclosing proprietary code and creating content that duplicates protected work.
• Tools like GitHub Copilot and ChatGPT lack the deep integration required for complicated coding activities, especially in game development sectors where platform-specific optimizations and interdependencies between code components are crucial.
• Research indicates that these techniques have variable success rates. ChatGPT generated accurate code for 65.2% of problems, followed by GitHub Copilot (46.3%) and Amazon CodeWhisperer (31.1%). Even the best-performing tool fails to produce accurate code in over one-third of circumstances.
• Furthermore, these tools' performance is strongly reliant on input quality. Without clear descriptions, their usefulness suffers dramatically.

On the other hand, OpenAI's new 'o1' series demonstrates considerable advancements in reasoning, maths, and coding abilities. These models outperform GPT-4 on complicated tasks, scoring 83% on International Mathematics Olympiad qualifying exams and achieving the 89th percentile in Codeforces coding competitions.

Conclusion: Balancing AI and Human Expertise

While AI models are improving and co-pilots help write some of the code, they do not replace the need for experienced coders. Human oversight is still required to ensure code correctness, maintain security, and manage large-scale projects. Developers that adopt and use these AI tools will have a significant advantage in an increasingly competitive sector.

As NVIDIA CEO Jensen Huang said:

“You’re not going to lose your job because of AI.

You’re going to lose your job because of someone else using AI.”

‍