Vidarc: Embodied Video Diffusion Model for Closed-loop Control

TL;DR

Vidarc tackles data-scarce robotic manipulation by uniting an embodied autoregressive video diffusion model with a masked inverse dynamics model to enable fast, grounded closed-loop control. The method trains with causal frame generation, an embodiment-aware diffusion loss, and a re-prefill inference strategy that leverages environment feedback and KV caching for low latency. Across large-scale cross-embodiment pretraining and finetuning on unseen platforms, Vidarc achieves higher real-world success, robust error correction, and dramatically reduced latency compared to strong baselines. These results demonstrate a scalable, transferable approach to embodied video learning that can operate in dynamic, real-world robotics settings with limited task-specific data.

Abstract

Robotic arm manipulation in data-scarce settings is a highly challenging task due to the complex embodiment dynamics and diverse contexts. Recent video-based approaches have shown great promise in capturing and transferring the temporal and physical interactions by pre-training on Internet-scale video data. However, such methods are often not optimized for the embodiment-specific closed-loop control, typically suffering from high latency and insufficient grounding. In this paper, we present Vidarc (Video Diffusion for Action Reasoning and Closed-loop Control), a novel autoregressive embodied video diffusion approach augmented by a masked inverse dynamics model. By grounding video predictions with action-relevant masks and incorporating real-time feedback through cached autoregressive generation, Vidarc achieves fast, accurate closed-loop control. Pre-trained on one million cross-embodiment episodes, Vidarc surpasses state-of-the-art baselines, achieving at least a 15% higher success rate in real-world deployment and a 91% reduction in latency. We also highlight its robust generalization and error correction capabilities across previously unseen robotic platforms.

Figures (8)

• Figure 1: Left: Vidarc consists of an embodied autoregressive video diffusion model and a masked inverse dynamics model. To enable closed-loop control, the inference pipeline re-prefills environment feedback into the autoregressive video generation. Right: After being pre-trained on approximately one million bimanual demonstration episodes, Vidarc is fine-tuned on an unseen platform using calibration with embodiment-specific masks; it achieves state-of-the-art performance and exhibits robust error correction capabilities.
• Figure 2: Vidarc comprises a video diffusion transformer and a masked inverse dynamics model. The video diffusion transformer is trained via teacher-forcing to predict the next observation based on previous observations and language instructions, while the masked inverse dynamics model is trained to infer actions from observations using a learnable masking mechanism that focuses attention on action-relevant regions. The learned mask is also used to reweight the diffusion loss, enhancing the video model’s focus on regions important for action prediction.
• Figure 3: Video predictions often get artifacts around the robot arm, which affects the task success.
• Figure 4: Video predictions, corresponding masks, and executions of Vidarc for dynamic tasks, where its error correction ability is observed.
• Figure 5: Execution of the method with closed-loop feedback. At frame 47, the model was grounded with real-world sensory data to correct generative drift, ensuring successful task execution.