A Survey of Interactive Generative Video

TL;DR

This survey defines Interactive Generative Video (IGV) and presents a five-module framework (Generation, Control, Memory, Dynamics, Intelligence) to unify development across gaming, embodied AI, and autonomous driving. It surveys foundational video-generation models (VAE, GAN, diffusion, autoregressive, and hybrids), and maps them onto IGV’s modules to highlight current capabilities and bottlenecks. The paper details domain-specific IGV applications, articulating challenges from real-time streaming and open-domain control to sim-to-real transfer and multimodal integration, and it outlines a path toward self-evolving virtual worlds. By providing a structured analysis and future directions, it aims to accelerate research and practical deployment of interactive, intelligent video systems. Overall, IGV is positioned as a versatile, data-driven platform for realistic, controllable video synthesis that can transform training, planning, and content creation across multiple high-stakes domains.

Abstract

Interactive Generative Video (IGV) has emerged as a crucial technology in response to the growing demand for high-quality, interactive video content across various domains. In this paper, we define IGV as a technology that combines generative capabilities to produce diverse high-quality video content with interactive features that enable user engagement through control signals and responsive feedback. We survey the current landscape of IGV applications, focusing on three major domains: 1) gaming, where IGV enables infinite exploration in virtual worlds; 2) embodied AI, where IGV serves as a physics-aware environment synthesizer for training agents in multimodal interaction with dynamically evolving scenes; and 3) autonomous driving, where IGV provides closed-loop simulation capabilities for safety-critical testing and validation. To guide future development, we propose a comprehensive framework that decomposes an ideal IGV system into five essential modules: Generation, Control, Memory, Dynamics, and Intelligence. Furthermore, we systematically analyze the technical challenges and future directions in realizing each component for an ideal IGV system, such as achieving real-time generation, enabling open-domain control, maintaining long-term coherence, simulating accurate physics, and integrating causal reasoning. We believe that this systematic analysis will facilitate future research and development in the field of IGV, ultimately advancing the technology toward more sophisticated and practical applications.

Figures (8)

• Figure 1: Evolutionary tree of Interactive Generative Video (IGV) models from 2020 to 2025. This diagram categorizes the development of IGV research from three major application domains: Game Simulation, Embodied AI, and Autonomous Driving, each represented by a colored trunk.
• Figure 2: Overview figure for GAN, VAE, Diffusion, and Autogression methods.
• Figure 3: Proposed framework of Interactive Generative Video (IGV). This figure illustrates the IGV system, which serves as a bridge between the real world and a virtual environment. In the real world, various roles such as players, designers, artists, and intelligent agents (e.g., robots, vehicles) interact with the IGV system through actions, instructions, and visual inputs. These diverse interactions naturally lead to applications in their respective domains: players engage with gaming applications, robots utilize embodied AI simulations, and vehicles operate in autonomous driving scenarios. The IGV system generates video outputs through five interconnected modules: the Generation module serves as the foundation for video synthesis; the Control module enables precise manipulation of video content; the Memory module maintains content consistency; the Dynamics module ensures physical plausibility; and the Intelligence module represents higher-level capabilities like causal reasoning. These modules work in concert to create and manage immersive virtual experiences through video generation.
• Figure 4: Overview figure of GameGAN gamegan. GameGAN is composed of three main modules. The dynamics engine (which refers to the internal mechanism that captures and updates the game's state transitions over time, simulating how the game world evolves in response to inputs gamegan) is implemented as an RNN and contains the world state updated at each time t. Optionally, it can write to and read from the external memory module. Finally, the rendering engine is used to decode the output image.
• Figure 5: Open-domain generation showcases from gamegen-x. GameGen-X enables high-fidelity and diverse generation of open-domain video game scenes, supporting various styles, characters, and virtual environments with cinematic quality.