Point Bridge: 3D Representations for Cross Domain Policy Learning

TL;DR

Point Bridge tackles the sim-to-real gap in robotic manipulation by casting scenes into a unified 3D point-based representation learned from synthetic data. It integrates a VLM-guided point extraction pipeline, MimicGen-based synthetic data expansion, and a transformer-based multitask policy to enable zero-shot transfer and scalable cross-task learning, with further gains when real demonstrations are included. The approach demonstrates robust zero-shot transfer across object instances and tasks, resilience to background clutter and viewpoint changes, and strong improvement when co-trained with modest real data. Depth estimation via Foundation Stereo and camera-aligned point sampling are key to reliable real-world performance, while ablations validate design choices and flexibility across sensing modalities. Collectively, Point Bridge reduces dependence on large real-world datasets and supports efficient, cross-domain policy learning with reproducible results and publicly releaseable resources.

Abstract

Robot foundation models are beginning to deliver on the promise of generalist robotic agents, yet progress remains constrained by the scarcity of large-scale real-world manipulation datasets. Simulation and synthetic data generation offer a scalable alternative, but their usefulness is limited by the visual domain gap between simulation and reality. In this work, we present Point Bridge, a framework that leverages unified, domain-agnostic point-based representations to unlock synthetic datasets for zero-shot sim-to-real policy transfer, without explicit visual or object-level alignment. Point Bridge combines automated point-based representation extraction via Vision-Language Models (VLMs), transformer-based policy learning, and efficient inference-time pipelines to train capable real-world manipulation agents using only synthetic data. With additional co-training on small sets of real demonstrations, Point Bridge further improves performance, substantially outperforming prior vision-based sim-and-real co-training methods. It achieves up to 44% gains in zero-shot sim-to-real transfer and up to 66% with limited real data across both single-task and multitask settings. Videos of the robot are best viewed at: https://pointbridge3d.github.io/

Figures (5)

• Figure 1: Point Bridge Overview. We present Point Bridge, a framework that leverages unified, domain-agnostic point-based representations to unlock the potential of large-scale synthetic simulation datasets. Point Bridge enables zero-shot sim-to-real policy transfer with minimal visual or object alignment, supports multitask learning, and further improves performance when co-trained with small amounts of real robot data.
• Figure 2: Point Extraction Pipeline Overview. Given a scene image and task description, Gemini team2023gemini identifies the task-relevant objects, which are then localized using Molmo deitke2024molmo and SAM-2 sam2 Subsequently, 3D keypoints on these objects are generated by uniformly sampling 2D keypoints on the image and projecting them into 3D using depth from Foundation Stereo wen2025foundationstereo, together with camera intrinsics and extrinsics.
• Figure 3: Tasks. Real-world rollouts showing Point Bridge's ability on 6 real-world tasks.
• Figure 4: Examples of background distractors in real-robot setup.
• Figure 5: Examples of failure cases of the VLM-guided scene filtering pipeline.