TC-IDM: Grounding Video Generation for Executable Zero-shot Robot Motion

TL;DR

This paper tackles the challenge of translating pixel-level, world-model plans into physically executable robot actions. It introduces Tool-Centric Inverse Dynamics Model (TC-IDM), a two-stage plan-and-translate framework that anchors control to the imagined end-effector trajectory while decoupling vision-driven state generation from geometry-guided gesture generation. By extracting dense 3D tool trajectories from generated videos and recovering 6-DoF end-effector motions via rigid-body constraints, TC-IDM achieves robust, viewpoint-invariant control across long-horizon tasks and deformable objects, markedly outperforming end-to-end VLA baselines and other IDMs. Real-world experiments show TC-IDM attaining 61.11% average success, with strong generalization across camera variations, deformable objects, and cross-embodiment transfers, validating its effectiveness as a scalable, generalizable bridge between visual foresight and physical manipulation.

Abstract

The vision-language-action (VLA) paradigm has enabled powerful robotic control by leveraging vision-language models, but its reliance on large-scale, high-quality robot data limits its generalization. Generative world models offer a promising alternative for general-purpose embodied AI, yet a critical gap remains between their pixel-level plans and physically executable actions. To this end, we propose the Tool-Centric Inverse Dynamics Model (TC-IDM). By focusing on the tool's imagined trajectory as synthesized by the world model, TC-IDM establishes a robust intermediate representation that bridges the gap between visual planning and physical control. TC-IDM extracts the tool's point cloud trajectories via segmentation and 3D motion estimation from generated videos. Considering diverse tool attributes, our architecture employs decoupled action heads to project these planned trajectories into 6-DoF end-effector motions and corresponding control signals. This plan-and-translate paradigm not only supports a wide range of end-effectors but also significantly improves viewpoint invariance. Furthermore, it exhibits strong generalization capabilities across long-horizon and out-of-distribution tasks, including interacting with deformable objects. In real-world evaluations, the world model with TC-IDM achieves an average success rate of 61.11 percent, with 77.7 percent on simple tasks and 38.46 percent on zero-shot deformable object tasks. It substantially outperforms end-to-end VLA-style baselines and other inverse dynamics models.

Figures (12)

• Figure 1: Tool-centric world-model-based IDM achieves stronger generalization than VLA baselines, providing a clean and rigid structure that is easier to learn and transfer across embodiments.
• Figure 2: Overview of Our Tool-Centric Video Generation Framework Given an initial RGB frame, depth map, and text instruction, our method utilizes a world model to generate high-level planning for extracting semantic vision features (DINOv3 simeoni2025dinov3 in our case), depth-aligned geometry(using Video Depth Anything chen2025video). We further employ a 3D motion tracker xiao2025spatialtrackerv2 to derive gripper-centric flow, guided jointly by the high-level plan and a gripper mask obtained via SAM 3 carion2025sam3segmentconcepts. Guided by the semantic vision feature and motion flow, the gripper and gesture states are respectively predicted via dedicated MLP heads.
• Figure 3: Representative manipulation tasks categorized by difficulty. The figure shows qualitative examples of tasks executed by the robot, grouped into three levels: Easy (top row), Medium (middle row), and Hard (bottom row). Easy tasks involve basic pick-and-place or push actions (e.g., placing bread on a plate); Medium tasks require more precise control or obstacle avoidance (e.g., taking a cup from a holder); Hard tasks involve fine-grained 6-DoF coordination (e.g., inserting chopsticks into a bamboo tube). This categorization provides a structured benchmark for evaluating the model's performance across spatial and temporal complexity.
• Figure 4: Long-Horizon Generalization.The world model generates a full six-step visual plan from the initial scene and the instruction “Fold the hoodie,” and the TC-IDM executes the plan in the real world, completing all stages of the folding sequence.
• Figure 5: Error Range Analysis. All methods receive the same world model-generated manipulation (top) and reproduce the action of placing the dart at the bullseye. The figure compares the execution accuracy of AVDCko2023learning, AnyPos tan2025anypos, and our TC-IDM.