Yume-1.5: A Text-Controlled Interactive World Generation Model

TL;DR

Yume1.5 tackles the challenge of real-time, text-controlled interactive world generation by introducing a three-pronged framework: Joint Temporal-Spatial-Channel Modeling (TSCM) for efficient long-video generation, a real-time acceleration strategy that eliminates KV caching via distillation and Self-Forcing, and a text-controlled world-event generation pathway trained with mixed datasets. The method integrates a dual-text-embedding scheme and keyboard-driven camera control to enable autoregressive, persistent exploration from single images or prompts. Empirical results demonstrate improved instruction following, stable long-video coherence, and practical generation speeds (e.g., ~12 fps at 540p on a single A100) compared with prior image-to-video and long-video baselines. These advances collectively enable more scalable, controllable, and interactive virtual world generation with potential applications in simulation, gaming, and embodied AI research.

Abstract

Recent approaches have demonstrated the promise of using diffusion models to generate interactive and explorable worlds. However, most of these methods face critical challenges such as excessively large parameter sizes, reliance on lengthy inference steps, and rapidly growing historical context, which severely limit real-time performance and lack text-controlled generation capabilities. To address these challenges, we propose \method, a novel framework designed to generate realistic, interactive, and continuous worlds from a single image or text prompt. \method achieves this through a carefully designed framework that supports keyboard-based exploration of the generated worlds. The framework comprises three core components: (1) a long-video generation framework integrating unified context compression with linear attention; (2) a real-time streaming acceleration strategy powered by bidirectional attention distillation and an enhanced text embedding scheme; (3) a text-controlled method for generating world events. We have provided the codebase in the supplementary material.

Figures (8)

• Figure 1: Our Yume1.5 framework supports three interactive generation modes: text-to-world generation from descriptions, image-to-world generation from static images, and text-based event editing. All modes are controlled through continuous keyboard inputs for person and camera movements, enabling autoregressive generation of explorable and persistent virtual worlds. We have included demo videos in the supplementary materials.
• Figure 2: An example of re-annotating the dataset. The original and new captions are used for T2V and I2V training, respectively. The Original caption describes detail scene context, while the New caption, generated by VLM, explicitly focuses on dynamic events.
• Figure 3: Core components of Yume1.5. (a) DiT Block with linear attention for efficient feature fusion. (b) Training pipeline with decomposed event and action descriptions. (c) Adaptive history tokens downsampling with varying compression rates based on temporal distance. (d) Chunk-based autoregressive inference with dual-compression memory management.
• Figure 4: Long-form video generation method. Left (Generator): The model autoregressively generates video chunks. Critically, it uses its own generated frames (rather than ground truth) as historical context—compressed by TSCM (Sec. 4.2)—to mitigate the train-inference discrepancy. Right (Distillation): The Fake Model (student) is optimized to match the trajectory of the Real Model (teacher) via a distribution matching gradient. This enables high-quality few-step inference while robustly handling error accumulation in long videos.
• Figure 5: Aesthetic Score Dynamics in Long-video Generation. Aesthetic Score Dynamics in Long-video Generation. The x-axis represents the number of video blocks (chronological segments), and the y-axis denotes the Aesthetic Score.