PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation

TL;DR

PointWorld advances 3D world modeling for robotic manipulation by unifying state and action into 3D point flows, enabling action-conditioned predictions from minimal perceptual input. Trained on ~2M trajectories across real and simulated environments, a single pre-trained checkpoint supports rigid, deformable, articulated, and tool-use tasks on a real robot with zero demonstrations, using only a single RGB-D image at deployment alongside model-predictive control. The work systematically studies backbone architectures, action representations, training objectives, partial observability, data mixtures, and scaling laws, yielding practical insights for scaling 3D world models. The proposed approach enables real-time planning and broad transfer across embodiments, highlighting a path toward general-purpose, perception-driven manipulation in unstructured settings. This has significant implications for robocontrol in the wild, reducing data collection and enabling versatile, cross-domain robotic capabilities.

Abstract

Humans anticipate, from a glance and a contemplated action of their bodies, how the 3D world will respond, a capability that is equally vital for robotic manipulation. We introduce PointWorld, a large pre-trained 3D world model that unifies state and action in a shared 3D space as 3D point flows: given one or few RGB-D images and a sequence of low-level robot action commands, PointWorld forecasts per-pixel displacements in 3D that respond to the given actions. By representing actions as 3D point flows instead of embodiment-specific action spaces (e.g., joint positions), this formulation directly conditions on physical geometries of robots while seamlessly integrating learning across embodiments. To train our 3D world model, we curate a large-scale dataset spanning real and simulated robotic manipulation in open-world environments, enabled by recent advances in 3D vision and simulated environments, totaling about 2M trajectories and 500 hours across a single-arm Franka and a bimanual humanoid. Through rigorous, large-scale empirical studies of backbones, action representations, learning objectives, partial observability, data mixtures, domain transfers, and scaling, we distill design principles for large-scale 3D world modeling. With a real-time (0.1s) inference speed, PointWorld can be efficiently integrated in the model-predictive control (MPC) framework for manipulation. We demonstrate that a single pre-trained checkpoint enables a real-world Franka robot to perform rigid-body pushing, deformable and articulated object manipulation, and tool use, without requiring any demonstrations or post-training and all from a single image captured in-the-wild. Project website at https://point-world.github.io/.

Figures (21)

• Figure 1: Overview of PointWorld. Given calibrated RGB-D, robot joint-space actions, and a robot description file (URDF), we convert actions to robot flows and concatenate with scene to form a single point cloud serving as an embodiment-agnostic interaction geometry. Scene points are featurized with a frozen DINOv3 encoder, robot points with temporal embeddings, and a point cloud backbone predicts full-scene 3D point flows.
• Figure 2: Rich Supervision of 3D World Modeling for Physical Interactions, when conditioned on 3D robot point flows and partial observable RGB-D. The 3D world modeling objective enjoys dense pixel-level supervision while encoding a wide range of capabilities central to robotic manipulation. To predict full-scene evolution, the model needs to implicitly segment objects of interest, identify material property and/or articulation structure, perform implicit shape completion for contact reasoning, propagate robot-object interaction for object-object dynamics, and simultaneously considering the effects of gravity, encapsulated all in a single forward pass of the learned model.
• Figure 2: Generalization of PointWorld across in-domain, cross-domain, held-out real environments under zero-shot and finetuned settings.$\text{D}$ denotes DROID, $\text{B}$ denotes B1K, $\text{H}$ denotes held-out real-world scenes. "From Scratch" denotes specialist trained on the held-out lab's data. Evaluations are done on unseen samples from the corresponding dataset. PointWorld generalizes within domains, zero-shot transfers to unseen real-world environments, surpasses specialists if finetuned with 20x fewer updates, and benefits from real-sim co-training.
• Figure 3: Movement Weighting and Uncertainty Regularization, where the robot releases and drops a yellow cloth. (Bottom Left) The movement weighting, used in the training objective, effectively biases the training towards scene points that are moving at each timestep, computed with the ground-truth flows. (Bottom Right) The uncertainty value, predicted by the model without any ground-truth, regularizes training to prevent overfitting to points that have unreliable ground-truth. Intriguingly, we observe that it also emerges to capture action-conditioned uncertainty arising from the object's physical properties (e.g., larger variability along the edge of the cloth).
• Figure 4: 3D Annotation Quality and Comparisons. FS denotes FoundationStereo foundationstereo; Dataset Extrinsics V1 and V2 are the two DROID extrinsics releases. (Top) Compared to DROID releases, our pipeline yields substantially higher-quality depth and camera pose calibration, resulting in more accurate robot mask overlays and better aligned point clouds (readers are encouraged to zoom in or check out the interactive visualization on the https://point-world.github.io for details). (Bottom) We further compute depth reprojection loss (differences between analytical and observed depth of robot surface), and F1 scores of point cloud alignment. We observe purely leveraging existing models (FS, VGGT) are insufficient, and V2 extrinsics improve over V1 by filtering out scenes with poor point cloud alignment but result in significantly lower scene counts. In contrast, our annotation pipeline retains substantially more scenes below $0.10$ depth-loss criterion and dominates all metrics.