51²è¹Ý Research Uses Novel Generative AI Framework to Push Boundaries of Procedural Narrative Generation, Partners to Fight Sex Trafficking
Dark Shadows
Achieving Next-Gen AI Storytelling Through Large Language Models
Developers: Zixin Zhaung, Tanishq Chawla, Jake Klinkert, Steph Buongiorno, Corey Clark
Experience a world shaped by your choices and AI in this immersive, film noir-style thriller. Be the detective in a procedurally generated mystery crafted by large language models (LLMs), where your actions shape an unfolding story.
Dark Shadows is a short video game that serves as a testbed for a novel, generative AI framework for procedural narrative generation. The game’s underlying components–used to dynamically generate story, assets, and in-game dialogue–were designed in a research lab at 51²è¹Ý Guildhall and published by a major academic conference, Artificial Intelligence in Interactive and Digital Entertainment (AIIDE).
A significant contribution is Dark Shadows’s underlying validation system, which pushes the boundaries of procedural narrative generation by enabling players to submit free-form text input (such as in conversation with an NPC) and receive an LLM-generated response that is aligned with the game narrative, even if the player submits off-topic or out-of-scope text. This feature tackles the challenge faced by many LLM-driven games today, where free-form text input can cause the game to derail.
Creating Dynamic Gameplay Content via Large Language Models
Research included in this project focused on developing a GEN-AI gaming framework named GAIA, designed to integrate Large Language Models (LLMs) for creating dynamic content in game development and gameplay. This backend framework allows developers to interact dynamically with LLMs, enabling personalized and real-time adaptive content across various gaming scenarios, enhancing both interactivity and player immersion.
Traditional game development has often relied on predetermined content and pathways, limiting the diversity and scalability of gameplay. By utilizing the GAIA framework, we can explore the potential of LLMs in game design to break away from conventional patterns, providing players with a richer and more dynamic gaming experience.
Over the course of this year-long research, team members designed and implemented several prototypes, testing the effectiveness of LLMs in generating tasks, dialogues, and narratives.
"This process not only sharpened my technical and project management skills but also deepened my understanding of the application of AI technology in game innovation," said programmer Zixin Zhuang, a Cohort 32 student at 51²è¹Ý Guildhall.
The findings from the GAIA framework offer game developers a novel tool that utilizes AI generative technology to enhance the dynamism and interactivity of game content. These insights can be applied to game level design, offering players a more free and personalized gaming experience, and opening up new possibilities for future game development.
Unveiling New Realms: Enhancing Procedural Narrative Generation and NPC Personalization using AI
The project aimed to develop a system design framework capable of training GPT-based agents. This work was intended to enhance the procedural narrative and character (NPC) personality generation models in video games.
According to team member and artist Tanishq Chawla, "Observing the limitations in current procedural narrative generation techniques, such as lack of emotional depth, lack of designer control and repetitiveness, [we] sought to create a more engaging and personalized gaming experience."
The project sought to integrate AI in a manner that allows for richer gameplay, enabling players to craft unique stories based on dynamic NPC interactions and evolving narratives. This was accomplished by employing a structured system design that guides the AI in generating coherent and contextually appropriate game narratives and character dialogues.
This project has the potential to revolutionize how stories are told within games. By improving narrative generation, GPT powered games can offer more diverse and engaging stories that adapt to player decisions in real-time, greatly enhancing player immersion and satisfaction.
PANGeA: Procedural Artificial Narrative using Generative AI for Turn-Based Video Games
This research introduces Procedural Artificial Narrative using Generative AI (PANGeA), a structured approach for leveraging large language models (LLMs), guided by a game designer's high-level criteria, to generate narrative content for turn-based role-playing video games (RPGs). Distinct from prior applications of LLMs used for video game design, PANGeA innovates by not only generating game level data (which includes, but is not limited to, setting, key items, and non-playable characters (NPCs)), but by also fostering dynamic, free-form interactions between the player and the environment that align with the procedural game narrative.
The NPCs generated by PANGeA are personality-biased and express traits from the Big 5 Personality Model in their generated responses. PANGeA addresses challenges behind ingesting free-form text input, which can prompt LLM responses beyond the scope of the game narrative. A novel validation system that uses the LLM's intelligence evaluates text input and aligns generated responses with the unfolding narrative. Making these interactions possible, PANGeA is supported by a server that hosts a custom memory system that supplies context for augmenting generated responses thus aligning them with the procedural narrative.
For its broad application, the server has a REST interface enabling any game engine to integrate directly with PANGeA, as well as an LLM interface adaptable with local or private LLMs. PANGeA's ability to foster dynamic narrative generation by aligning responses with the procedural narrative is demonstrated through an empirical study and ablation test of two versions of a demo game. These are, a custom, browser-based GPT and a Unity demo.
As the results show, PANGeA holds potential to assist game designers in using LLMs to generate narrative-consistent content even when provided varied and unpredictable, free-form text input.
World-Changing Real Applications: Using Gaming to Fight Sex Trafficking
A Framework for Leveraging Human Computation Gaming to Enhance Knowledge Graphs for Accuracy Critical Generative AI Applications
External knowledge graphs (KGs) can be used to augment large language models (LLMs), while simultaneously providing an explainable knowledge base of facts that can be inspected by a human. This approach may be particularly valuable in domains where explainability is critical, like human trafficking data analysis.
However, creating KGs can pose challenges. KGs parsed from documents may comprise explicit connections (those directly stated by a document) but miss implicit connections (those obvious to a human although not directly stated). To address these challenges, this preliminary research introduces the GAME-KG framework, standing for "Gaming for Augmenting Metadata and Enhancing Knowledge Graphs."
GAME-KG is a federated approach to modifying explicit as well as implicit connections in KGs by using crowdsourced feedback collected through video games. GAME-KG is shown through two demonstrations: a Unity test scenario from Dark Shadows, which collects feedback on KGs parsed from US Department of Justice (DOJ) Press Releases on human trafficking, and a following experiment where OpenAI's GPT-4 is prompted to answer questions based on a modified and unmodified KG. Initial results suggest that GAME-KG can be an effective framework for enhancing KGs, while simultaneously providing an explainable set of structured facts verified by humans.
Partnering with law enforcement, the 51²è¹Ý Research team's game Dark Shadows provides machine learning via gameplay as they work together to fight sex trafficking. As players explore this digital world to extract data and unfold clues in the game, they are thereby also working on real-life cases to parse the database of trafficking information, identify real traffickers, and connect patterns to help stop future trafficking.
Learn More
In the News:
- CBS feature —
- 51²è¹Ý feature —
- Dallas Innovates feature —
Published Articles:
Published Citations:
(Note: these exist in academic databases and are inaccessible to the public)
- Steph Buongiorno [Corresponding Author], Jake Klinkert, Tanishq Chawla, Zixin Zhuang, and Corey Clark. "PANGeA: Procedural Artificial Narrative Using Generative AI for Turn-Based Video Games." Proceedings of the 2024 AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), Lexington, KY, USA, 2024.
- Steph Buongiorno [Corresponding Author], and Corey Clark. "Leveraging Gaming to Enhance Knowledge Graphs for Explainable Generative AI Applications" Proceedings of the 2024 IEEE Conference on Games (CoG), Milan, Italy, 2024.
AI-Driven Human Trafficking Data Ecosystems
Developers: This project is a collaboration between 51²è¹Ý Guildhall (Video Games), Lyle School of Engineering, and Economics
To make the response to human trafficking more resilient, we propose a new data ecosystem of open knowledge networks (OKNs)–combined with information retrieval technology that is driven by AI agents–to improve data accessibility among diverse stakeholders. Our research will focus on the state of Texas and confront the problems surrounding the development and use of a distributed, open knowledge network that holds sensitive information that cannot be shared directly. The initial hub for this ecosystem will be 51²è¹Ý's Human Trafficking Data Warehouse, supported by funding from the National Institute of Justice (NIJ). In this phase of our project, we will research, design, and develop a scalable, AI agent-based OKN architecture to facilitate information flow between stakeholders such as researchers, law enforcement, DHS, and travel intermediaries. This system will enable users to input knowledge artifacts and interact with an AI-driven interface that leverages the OKN to automatically identify relevant data and techniques to answer their queries. Our goal is to improve the information exchange between these stakeholders while adhering to legal standards and minimizing the risks to individual and societal harm, safeguarding civil rights, and protecting marginalized populations.
Future Impacts: The proposed AI-driven open knowledge network (OKN) has the potential to transform how stakeholders combat human trafficking by improving collaboration, decision-making, and data accessibility while safeguarding sensitive information. Focused initially on Texas, the scalable system would enable seamless information exchange among researchers, law enforcement, and other entities, ensuring privacy and compliance with civil rights. By leveraging AI to identify relevant data and techniques, the OKN could enhance intervention efforts, empower marginalized communities, and set a precedent for ethical AI use in addressing systemic social challenges.
Figure 1: Key to this research are: (a) the design of human-in-the-loop mechanics that enable users to contribute knowledge to the data ecosystem; and (b) “knowledge graphs” – an underlying data structure that can be used to store information about entities and their relationships. This work builds upon our previous research that uses video games as an interface to collect human feedback and augment knowledge bases. Here, the player is presented with “detective notes.” The player must scan these notes for data features that can be used to build knowledge graphs, representing information about human trafficking.
Figure 2: An example of a knowledge graph. Knowledge graphs can be automatically parsed from text data, however, they may not contain all the information represented by the document. The user’s input can be used to connect additional entities (represented by the dotted lines).