Causal World Modeling for Robot Control

TL;DR

LingBot-VA presents an autoregressive diffusion world model that jointly predicts visual dynamics and infers actions for robotic manipulation. By interleaving video and action tokens within a single causal Mixture-of-Transformers backbone and enabling closed-loop integration of real-world observations, it achieves state-of-the-art performance on long-horizon, precision, and deformable-object tasks in both simulation and real-world deployments. The approach introduces Noisy History Augmentation, KV-cache based memory, and an asynchronous inference pipeline to maintain high-frequency control with reduced latency. Extensive ablations and cross-domain evaluations demonstrate strong data efficiency, temporal memory, and generalization to novel objects and configurations, with public code and checkpoints to support reproducibility.

Abstract

This work highlights that video world modeling, alongside vision-language pre-training, establishes a fresh and independent foundation for robot learning. Intuitively, video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics. Inspired by this, we introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously. Our model features three carefully crafted designs: (1) a shared latent space, integrating vision and action tokens, driven by a Mixture-of-Transformers (MoT) architecture, (2) a closed-loop rollout mechanism, allowing for ongoing acquisition of environmental feedback with ground-truth observations, (3) an asynchronous inference pipeline, parallelizing action prediction and motor execution to support efficient control. We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations. The code and model are made publicly available to facilitate the community.

Figures (10)

• Figure 1: LingBot-VA: An Autoregressive World Model for Robotic Manipulation. (1) Pretraining:LingBot-VA is pretrained on diverse in-the-wild videos and robot action data, enabling strong generalization across scenes and objects. (2) Comprehensive Evaluation: We conduct extensive experiments on real-world tasks (long-horizon, deformable objects, and precision manipulation) and simulation benchmarks, significantly outperforming state-of-the-art methods including $\pi_{0.5}$. (3) Versatile Capabilities: Beyond policy learning, our model supports visual dynamics prediction and inverse dynamics inference from robot videos. (4) Emergent Properties: Our causal world modeling approach exhibits long-range temporal memory and strong few-shot adaptation ability.
• Figure 2: Framework overview: LingBot-VA is conditioned by autoregressive diffusion for unified video-action world modeling. We leverage a dual-stream Mixture-of-Transformers (MoT) architecture that interleaves video and action tokens within a single sequence. At each autoregressive step, the video stream (initialized from Wan2.2-5B) first predicts future latent visual states via flow matching. Then the action stream decodes corresponding actions through inverse dynamics conditioning on the predicted visual transitions.
• Figure 3: Teacher Forcing Attention Mask: Causal attention mask for unified video-action pretraining. Each token can only attend to preceding tokens in the temporal sequence.
• Figure 4: Asynchronous pipeline design overview: The traditional synchronous pipeline (A) suffers from delays caused by blocked computations, while the asynchronous pipeline (B) addresses this issue by enabling parallel computation and execution. However, a naive asynchronous implementation (B-1) relies on outdated visual predictions. In contrast, we improve and refine asynchronous prediction through forward dynamic prediction (B-2), which updates stale predictions with recent real-world observations.
• Figure 5: Real-world deployment results. We evaluate LingBot-VA on six manipulation tasks across three categories: long-horizon tasks (Make Breakfast, Pick Screws), precision tasks (Insert Tubes, Unpack Delivery), and deformable & articulated object manipulation (Fold Clothes, Fold Pants). Our method achieves state-of-the-art performance on both metrics.