Geometry-aware 4D Video Generation for Robot Manipulation

TL;DR

This work tackles the challenge of generating temporally coherent and geometrically consistent 4D RGB-D videos for robotic manipulation. It introduces geometry-consistent supervision that enforces cross-view pointmap alignment, learning a shared 3D scene representation to generate future views from novel viewpoints without camera poses. By combining a diffusion-based video backbone with dual-view decoders and cross-attention, the approach achieves improved multi-view consistency and depth accuracy, and facilitates end-effector trajectory recovery via 6DoF pose tracking. Experiments on simulated and real-world tasks show superior video quality and cross-view alignment, with notable gains in downstream manipulation success. The method enables flexible camera setups and robust visuomotor generalization, advancing perception-driven robotics in multi-view settings.

Abstract

Understanding and predicting the dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of videos by supervising the model with cross-view pointmap alignment during training. This geometric supervision enables the model to learn a shared 3D representation of the scene, allowing it to predict future video sequences from novel viewpoints based solely on the given RGB-D observations, without requiring camera poses as inputs. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, supporting robust robot manipulation and generalization to novel camera viewpoints.

Figures (10)

• Figure 1: Geometry-aware 4D Video Generation. Our model takes RGB-D observations from two camera views and predicts future 4D pointmaps in the coordinate frame of the reference view $v_n$. The blue pointmap is predicted from camera $v_n$, while the red pointmap shows the prediction from camera $v_m$ projected into the coordinate frame of $v_n$. RGB videos are predicted separately for each view. Together, the model enables geometry-consistent 4D video generation.
• Figure 2: 4D Video Generation for Robot Manipulation. Our model takes RGB-D observations from two camera views, and predicts future pointmaps and RGB videos. To ensure cross-view consistency, we apply cross-attention in the U-Net decoders for pointmap prediction. The resulting 4D video can be used to extract the 6DoF pose of the robot end-effector using pose tracking methods, enabling downstream manipulation tasks.
• Figure 3: Robot Manipulation Tasks in Simulation.
• Figure 4: Qualitative Results and Comparisons under Novel Camera Views. Our method generates geometrically consistent 4D videos across camera views. In contrast, baseline results often exhibit significant cross-view inconsistencies or contain noticeable artifacts in the RGB or depth predictions.
• Figure 5: Real World 4D Video Generation Results on PutSpatulaOnTable. Our model predicts high-fidelity RGB-D sequences that capture the robot gripper motions. In this particular sequence, the model correctly predicts how the robot reaches the spatula, grasps it, and lifts it up from the utensil crock.