GeoPredict: Leveraging Predictive Kinematics and 3D Gaussian Geometry for Precise VLA Manipulation

TL;DR

GeoPredict tackles the 3D reasoning gap in Vision-Language-Action robotic policies by introducing two training-time predictive priors: trajectory-level kinematic predictions and predictive 3D Gaussian geometry with track-guided refinement. These priors are integrated through a block-wise attention scheme and supervised via depth rendering, while inference remains lightweight. Empirical results on RoboCasa, LIBERO, and real-world tasks show substantial improvements in geometry-intensive and spatially demanding scenarios, validating the approach's effectiveness and practicality. The work advances grounded, predictive VLA control by coupling explicit 3D geometry with horizon-aware kinematics in a scalable, pretrained framework.

Abstract

Vision-Language-Action (VLA) models achieve strong generalization in robotic manipulation but remain largely reactive and 2D-centric, making them unreliable in tasks that require precise 3D reasoning. We propose GeoPredict, a geometry-aware VLA framework that augments a continuous-action policy with predictive kinematic and geometric priors. GeoPredict introduces a trajectory-level module that encodes motion history and predicts multi-step 3D keypoint trajectories of robot arms, and a predictive 3D Gaussian geometry module that forecasts workspace geometry with track-guided refinement along future keypoint trajectories. These predictive modules serve exclusively as training-time supervision through depth-based rendering, while inference requires only lightweight additional query tokens without invoking any 3D decoding. Experiments on RoboCasa Human-50, LIBERO, and real-world manipulation tasks show that GeoPredict consistently outperforms strong VLA baselines, especially in geometry-intensive and spatially demanding scenarios.

Figures (4)

• Figure 1: Overview of GeoPredict. Given an instruction, multi-view images and motion history encoded by the Track Encoder, a central LLM Transformer learns two main tasks. First, it predicts multi-timestep 3D keypoint trajectories using learnable Future Track Query. Second, it forecasts future workspace geometry as a predictive 3D Gaussian by processing a 3D Spatial Query through a Voxel Decoder. A track-guided refinement mechanism leverages the predicted future tracks to allocate geometric capacity to task-relevant interaction regions. Our policy then generates the final action via an Action Expert. Crucially, these predictive modules serve exclusively as training-time supervision and are not invoked during inference, thus preserving efficiency.
• Figure 2: Block-wise Causal Attention Mechanism. For simplicity, the detailed attention pathways from the 3D Token and State Token blocks to other blocks are not fully drawn.
• Figure 3: Real-world Evaluation Suite. These settings aim to evaluate the model's spatial generalization, geometry generalization and robustness to distractors. Each column represents different trials of the same task.
• Figure 4: Qualitative Comparisons of Future Depth Rendering. Visualizations are shown for timesteps $t+1, t+10, \text{and } t+20$. Red boxes highlight the improvements in fine-grained geometric details. Best viewed zoomed in.