BridgeV2W: Bridging Video Generation Models to Embodied World Models via Embodiment Masks

TL;DR

BridgeV2W tackles the misalignment between coordinate-space actions and pixel-space videos in embodied world models by converting actions into pixel-aligned embodiment masks rendered from URDF and camera parameters and conditioning a pretrained video generator via a ControlNet-style pathway. A flow-based motion loss further concentrates supervision on task-relevant motion, improving dynamic region learning and temporal coherence. The approach achieves stronger video generation quality and view robustness on single-arm and dual-arm robots, while enabling downstream policy evaluation and goal-conditioned planning with a unified, cross-embodiment framework. By leveraging pretrained visual and motion priors and accommodating uncalibrated data, BridgeV2W offers scalable, versatile embodied world modeling applicable to real-world robotics tasks.

Abstract

Embodied world models have emerged as a promising paradigm in robotics, most of which leverage large-scale Internet videos or pretrained video generation models to enrich visual and motion priors. However, they still face key challenges: a misalignment between coordinate-space actions and pixel-space videos, sensitivity to camera viewpoint, and non-unified architectures across embodiments. To this end, we present BridgeV2W, which converts coordinate-space actions into pixel-aligned embodiment masks rendered from the URDF and camera parameters. These masks are then injected into a pretrained video generation model via a ControlNet-style pathway, which aligns the action control signals with predicted videos, adds view-specific conditioning to accommodate camera viewpoints, and yields a unified world model architecture across embodiments. To mitigate overfitting to static backgrounds, BridgeV2W further introduces a flow-based motion loss that focuses on learning dynamic and task-relevant regions. Experiments on single-arm (DROID) and dual-arm (AgiBot-G1) datasets, covering diverse and challenging conditions with unseen viewpoints and scenes, show that BridgeV2W improves video generation quality compared to prior state-of-the-art methods. We further demonstrate the potential of BridgeV2W on downstream real-world tasks, including policy evaluation and goal-conditioned planning. More results can be found on our project website at https://BridgeV2W.github.io .

Figures (12)

• Figure 1: BridgeV2W vs. previous methods. Pixel-aligned embodiment masks bridge video generation models to embodied world models, addressing the action–video gap, improving viewpoint robustness, and yielding a unified architecture across embodiments.
• Figure 2: Overview of the BridgeV2W pipeline. Actions are projected into pixel-space masks using URDF and camera parameters. The initial image and mask sequence are encoded by VAE, with mask features injected via a ControlNet branch into the DiT backbone. The model generates action-consistent videos, trained with diffusion, dynamics-consistency, and flow-based objectives.
• Figure 3: Qualitative video generation results produced by BridgeV2W.
• Figure 4: Correlation between BridgeV2W policy evaluation and real-world success rates.
• Figure 5: The visualization of the URDF of Franka Panda and AgiBot G1.