Yan: Foundational Interactive Video Generation

TL;DR

Yan introduces a foundational, end-to-end framework for interactive video generation that unifies high-fidelity real-time simulation (Yan-Sim), prompt-controllable multi-modal generation (Yan-Gen), and on-the-fly multi-granularity editing (Yan-Edit) built on a large, annotated 3D-game dataset. The approach combines a high-compression 3D-VAE with shift-window denoising for 1080P/60FPS simulation, hierarchical world/local captions plus auto-regressive/post-training distillation for real-time generation, and a depth-based editing pipeline that separately handles mechanics and rendering. Key contributions include a scalable data-collection pipeline (400M frames across 90 styles), a real-time diffusion-based generation pipeline with cross-domain generalization, and interactive editing capable of structure and style changes during playback. Yan demonstrates how integrated simulation, generation, and editing can enable open-domain, AI-driven interactive media tools and sets a path toward scalable, interactive AI worlds.

Abstract

We present Yan, a foundational framework for interactive video generation, covering the entire pipeline from simulation and generation to editing. Specifically, Yan comprises three core modules. AAA-level Simulation: We design a highly-compressed, low-latency 3D-VAE coupled with a KV-cache-based shift-window denoising inference process, achieving real-time 1080P/60FPS interactive simulation. Multi-Modal Generation: We introduce a hierarchical autoregressive caption method that injects game-specific knowledge into open-domain multi-modal video diffusion models (VDMs), then transforming the VDM into a frame-wise, action-controllable, real-time infinite interactive video generator. Notably, when the textual and visual prompts are sourced from different domains, the model demonstrates strong generalization, allowing it to blend and compose the style and mechanics across domains flexibly according to user prompts. Multi-Granularity Editing: We propose a hybrid model that explicitly disentangles interactive mechanics simulation from visual rendering, enabling multi-granularity video content editing during interaction through text. Collectively, Yan offers an integration of these modules, pushing interactive video generation beyond isolated capabilities toward a comprehensive AI-driven interactive creation paradigm, paving the way for the next generation of creative tools, media, and entertainment. The project page is: https://greatx3.github.io/Yan/.

Figures (16)

• Figure 1: Comprehensive capabilities of Yan. Yan supports real-time interactive video generation, with all interactions driven by user input. It offers a wide range of capabilities, including AAA-level simulations, multi-modal generation, and multi-granularity editing. Notably, during the editing process, users can dynamically modify prompts to edit subsequent generated content interactively.
• Figure 2: The overall framework of Yan. First, we leverage the agent to collect and clean data in the game environment. Next, we use a vision language model and a depth estimation model to annotate the collected data, generating a structured text prompt and depth. Finally, both the labeled data and the unlabeled open-domain data are used together for training.
• Figure 3: The overview of the data collection pipeline.
• Figure 4: The overall framework of Yan-Sim. The top part of the image depicts Yan-Sim's entire inference flow, where init frames are only provided during initial inference.
• Figure 5: The visualization results of visual quality using Yan-Sim. Yan-Sim possesses the ability to simulate diverse styles and interactive scenarios.