Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos

TL;DR

MTV-World is proposed, an embodied world model that introduces Multi-view Trajectory-Video control for precise visuomotor prediction and achieves precise control execution and accurate physical interaction modeling in complex dual-arm scenarios.

Abstract

Embodied world models aim to predict and interact with the physical world through visual observations and actions. However, existing models struggle to accurately translate low-level actions (e.g., joint positions) into precise robotic movements in predicted frames, leading to inconsistencies with real-world physical interactions. To address these limitations, we propose MTV-World, an embodied world model that introduces Multi-view Trajectory-Video control for precise visuomotor prediction. Specifically, instead of directly using low-level actions for control, we employ trajectory videos obtained through camera intrinsic and extrinsic parameters and Cartesian-space transformation as control signals. However, projecting 3D raw actions onto 2D images inevitably causes a loss of spatial information, making a single view insufficient for accurate interaction modeling. To overcome this, we introduce a multi-view framework that compensates for spatial information loss and ensures high-consistency with physical world. MTV-World forecasts future frames based on multi-view trajectory videos as input and conditioning on an initial frame per view. Furthermore, to systematically evaluate both robotic motion precision and object interaction accuracy, we develop an auto-evaluation pipeline leveraging multimodal large models and referring video object segmentation models. To measure spatial consistency, we formulate it as an object location matching problem and adopt the Jaccard Index as the evaluation metric. Extensive experiments demonstrate that MTV-World achieves precise control execution and accurate physical interaction modeling in complex dual-arm scenarios.

Figures (8)

• Figure 1: Overview of the proposed MTV-World, which utilizes multi-view trajectory videos as control inputs. Each trajectory video is synthesized by combining the initial object mask with trajectory motions for future frame prediction. The multi-view trajectories clearly depict the robot arm’s movement paths and its interactions with objects.
• Figure 2: Illustration of the MTV-World framework. (a) Trajectory and object representation: trajectory control videos are generated by combining object masks and the trajectories of luminous points on the image to serve as control inputs. (b) MTV-World architecture: the model takes multi-view video sequences and trajectory control videos as inputs. The first frame is used as a reference image, which is simultaneously processed by CLIP for semantic encoding and by a shared VAE encoder to obtain reference latents, which are later removed before decoding. (c) Automated evaluation pipeline: performance is assessed by measuring the spatial alignment between the predicted and ground-truth object masks in videos using the Jaccard Index.
• Figure 3: Success rate of object interactions over rollout progress. The x-axis denotes the rollout progress percentage, while the y-axis represents the success rate of object interactions measured by the $\mathcal{J}$ metric.
• Figure 4: Qualitative comparison with baseline methods, where two representative examples: a successful Stack Blocks task (top) and a failed rollout of the Place Bread Plate task (bottom) are presented.
• Figure 5: Visualization of the generated multi-view trajectory videos, where two examples: Shake Bottle (top) and Collect Toy (bottom), are shown.