TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model

TL;DR

TeleWorld presents a real-time 4D world model that unifies video generation, dynamic 3D reconstruction, and persistent memory within a closed-loop system. It introduces a generation-reconstruction-guidance loop and a Macro-from-Micro Planning (MMPL) framework with Distribution Matching Distillation (DMD) to achieve long-horizon, memory-aware synthesis under practical compute budgets. The approach is complemented by real-time 4D reconstruction (including key-frame reconstruction and moving-object segmentation), keyboard/view-conditioned guidance, a streaming diffusion-based pipeline with a Stream-VAE and online video super-resolution, and a scalable training system enabling real-time diffusion distillation on large models. Extensive experiments on TeleWorld-500K and the WorldScore benchmark demonstrate strong static and dynamic performance, robust long-term consistency, and real-time generation efficiency, marking TeleWorld as a practical foundation for interactive, memory-enabled multimodal world models and embodied intelligence.

Abstract

World models aim to endow AI systems with the ability to represent, generate, and interact with dynamic environments in a coherent and temporally consistent manner. While recent video generation models have demonstrated impressive visual quality, they remain limited in real-time interaction, long-horizon consistency, and persistent memory of dynamic scenes, hindering their evolution into practical world models. In this report, we present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system. TeleWorld introduces a novel generation-reconstruction-guidance paradigm, where generated video streams are continuously reconstructed into a dynamic 4D spatio-temporal representation, which in turn guides subsequent generation to maintain spatial, temporal, and physical consistency. To support long-horizon generation with low latency, we employ an autoregressive diffusion-based video model enhanced with Macro-from-Micro Planning (MMPL)--a hierarchical planning method that reduces error accumulation from frame-level to segment-level-alongside efficient Distribution Matching Distillation (DMD), enabling real-time synthesis under practical computational budgets. Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible systems. Extensive experiments demonstrate that TeleWorld achieves strong performance in both static and dynamic world understanding, long-term consistency, and real-time generation efficiency, positioning it as a practical step toward interactive, memory-enabled world models for multimodal generation and embodied intelligence.

Figures (4)

• Figure 1: Structure of TeleWorld. We propose a dynamic "Generation-Reconstruction-Guidance" closed-loop framework for 4D spatio-temporal modeling. The model first generates an initial set of videos based on the user’s pre-defined instructions. It then enters a loop where, in each iteration, it processes the user’s real-time input instructions, reconstructs the video output from the previous round, and renders it according to the input camera poses. The rendered results serve as guidance to direct the current round of video generation and motion synthesis, and this process repeats iteratively.
• Figure 2: Our macro-from-micro planning framework is organized into two levels: (1) Micro Planning, where a sequence of frames is generated within each local segment to constrain error propagation; and (2) Macro Planning, which links segments through an autoregressive chain—each step’s output frames guide the prediction of the next, ensuring long-range temporal consistency. As shown in the figure, the three predicted frames marked in green correspond to the initial pre-planning frames, $\mathcal{P}_{\mathcal{M}_s} = \{x_s^{t_a}, x_s^{t_b}, x_s^{t_c}\}$ , which serve as keyframes to maintain long-term memory and stability throughout the video sequence.
• Figure 3: Pipeline execution schedules for Distribution-Matching Distillation. (a) Generator-step pipeline with 7 micro-batches. Cell length denotes execution time. The critic and teacher works in parallel, so their cells are merged together for simplicity, and their cell length denotes the maximum of their execution time. The upper half of the figure is the non-pipelining baseline, which introduces a lot of GPU bubbles (i.e. GPU idle time). The lower half is our proposed pipeline schedule. In the stable phase, the generator backward stage of micro-batch $i$ and the generator forward stage of micro-batch $i+2$ are executed concurrently with the critic/teacher forward stage of micro-batch $i+1$. The execution time of all stages are carefully balanced by allocating appropriate numbers of GPUs to each component, enabling near-perfect overlap. This method minimizes GPU bubbles and achieves efficient parallelization of generator, teacher, and critic workloads in the proposed system. (b) Critic-step pipeline with 4 micro-batches. Since the generator parameters remain frozen during the critic update, the pipeline follows a simpler producer-consumer execution pattern.
• Figure 4: Multi-GPU parallel inference via adaptive workload scheduling. Given the initial frame $f^0_1$, segment 0 first generates its planning frames $f^0_2$, $f^0_6$, and $f^0_{10}$. These planning frames then guide the content population of the intermediate frames $f^0_3$, $f^0_4$, and $f^0_5$. While segment 0 is still populating these frames, segment 1 can immediately start its Micro Planning by taking $f^0_{10}$ as the initial frame $f^1_1$ and generating its own planning frames $f^1_2$, $f^1_6$, and $f^1_{10}$. This staged execution enables overlapping planning and populating across segments, maximizing multi-GPU parallelism. Here, each $t_i$ denotes an inference step in the diffusion sampling process.