A Platform for Interactive AI Character Experiences

TL;DR

The paper tackles the challenge of delivering believable, story driven interactive characters by proposing a modular platform that unifies conversational AI, memory, personality, voice synthesis, animation, and real_world sensing within a Unity-based system. A proof_of_concept, Digital Einstein, demonstrates the architecture by combining GPT_4o and a fine_tuned Llama 3 8B with retrieval_augmented memory, embedding driven topic steering, emotion aware synthesis, image generation, and environment awareness. Key contributions include a scalable, extensible pipeline with dynamic personality control, memory for coherence, multimodal perception, and privacy aware operation, enabling real_time, immersive interactions in a physical setup. The work shows promising results in real_world deployments, achieving responsive conversations, coherent topic flow, and visually enriched narratives, and it outlines concrete future directions for latency reduction, richer animation, and more systematic user studies to quantify component contributions.

Abstract

From movie characters to modern science fiction - bringing characters into interactive, story-driven conversations has captured imaginations across generations. Achieving this vision is highly challenging and requires much more than just language modeling. It involves numerous complex AI challenges, such as conversational AI, maintaining character integrity, managing personality and emotions, handling knowledge and memory, synthesizing voice, generating animations, enabling real-world interactions, and integration with physical environments. Recent advancements in the development of foundation models, prompt engineering, and fine-tuning for downstream tasks have enabled researchers to address these individual challenges. However, combining these technologies for interactive characters remains an open problem. We present a system and platform for conveniently designing believable digital characters, enabling a conversational and story-driven experience while providing solutions to all of the technical challenges. As a proof-of-concept, we introduce Digital Einstein, which allows users to engage in conversations with a digital representation of Albert Einstein about his life, research, and persona. While Digital Einstein exemplifies our methods for a specific character, our system is flexible and generalizes to any story-driven or conversational character. By unifying these diverse AI components into a single, easy-to-adapt platform, our work paves the way for immersive character experiences, turning the dream of lifelike, story-based interactions into a reality.

Figures (15)

• Figure 1: System Overview: The pipeline processes sensor inputs, including transcribed speech and video-based user characteristics and behavior analysis, through an LLM-based chatbot supported by memory and an adjustable personality of the digital character. The chatbot's responses guide speech and facial animation synthesis based on emotions detected in the response, motion-captured body animation selected based on the avatar state, and image generation.
• Figure 2: Visualization of the embedding space from synthetic Einstein conversations. The clusters of blue points represent different topics. A user interaction trace is highlighted in red, with the starting point marked in green. The transition from the topic "Nobel Prize" to "childhood" is initiated by the user.
• Figure 3: Sliders for real-time personality adjustment across five traits: Vibrancy, Conscientiousness, Decency, Artificiality, and Neuroticism.
• Figure 4: Left: LOE distribution between retargeted motion-capture sequences and animations synthesized from the corresponding audio using Audio2Face (red, $\mu = 0.005$) and SALSA (blue, $\mu = 0.006$). Right: Motion-capture setup featuring eight positional trackers, two hand gesture trackers, and a full facial tracker mounted on a gimbal helmet.
• Figure 5: The system identifies the topic from the conversation and uses it to generate a matching image (e.g., a black hole).