Advancing Open-source World Models

TL;DR

LingBot-World presents an open-source world model framework that transitions from video-generation to interactive, long-horizon simulation. It builds a scalable Data Engine and a three-stage training pipeline (pre-training, middle-training, post-training) to achieve long-term memory, action controllability, and real-time inference, enabled by techniques such as MoE architectures, Plücker embeddings, AdaLN, block causal attention, and distribution matching distillation with adversarial augmentation. The system demonstrates emergent memory, ultra-long coherent generation, and practical applications in promptable events, agent learning, and 3D reconstruction, while acknowledging limitations in memory stability, computation, and interaction granularity. This work offers a practical open-source platform to democratize high-fidelity world modeling and interactive simulation for content creation, gaming, and robot learning.

Abstract

We present LingBot-World, an open-sourced world simulator stemming from video generation. Positioned as a top-tier world model, LingBot-World offers the following features. (1) It maintains high fidelity and robust dynamics in a broad spectrum of environments, including realism, scientific contexts, cartoon styles, and beyond. (2) It enables a minute-level horizon while preserving contextual consistency over time, which is also known as "long-term memory". (3) It supports real-time interactivity, achieving a latency of under 1 second when producing 16 frames per second. We provide public access to the code and model in an effort to narrow the divide between open-source and closed-source technologies. We believe our release will empower the community with practical applications across areas like content creation, gaming, and robot learning.

Figures (16)

• Figure 1: Interactive world simulation across diverse environments. The figure showcases selected samples generated by LingBot-World, demonstrating its capability to synthesize high-fidelity videos in various domains, including photorealistic landscapes, scientific visualizations, and stylized artistic worlds. The overlaid keyboard icons (W, A, S, D) highlight the model's controllability, allowing users to navigate and interact with these dynamic environments seamlessly.
• Figure 2: Game and synthetic data acquisition. The system leverages computational resources and software platforms to capture visual observations that are temporally aligned with action signals and camera states.
• Figure 3: Overview of data profiling engine. The process bridges the gap between raw video collections and training-ready assets. It integrates physical attribute filtering, semantic profiling, and geometric annotation to establish a robust foundation for the subsequent hierarchical captioning generation.
• Figure 4: Overview of the LingBot-World training pipeline. We propose a multi-stage evolution strategy to transform a foundation video generator into an interactive world simulator. Pre-training stage establishes a robust general video prior to ensure high-fidelity open-domain generation. Middle-training stage injects world knowledge and action controllability, enabling the model to simulate long-term dynamics with consistent interactive logic. Post-training stage adapts the architecture for real-time interaction, employing causal attention and few-step distillation to achieve low latency and strict causality.
• Figure 5: Pipeline of LingBot-World. The left part shows the pipeline of LingBot-World video generation. LingBot-World uses an image or a video, noisy latents, and user-defined action signals as inputs to generate video sequences with spatial memory, long-term consistency, and precise action following. The right part shows the architecture of the DiT blocks in LingBot-World. The video latent first passes through a self-attention layer, enabling LingBot-World to learn spatiotemporal coherence, and further emerge spatial memory ability. Then, action signals are injected through a Plücker Encoder, where the input actions are projected into Plücker embeddings and modulate the video latent via adaptive normalization that transforms the Plücker embeddings into scaling and shifting factors. Finally, a cross-attention layer is applied to condition the video latent on text embeddings.