Learning Geometrically-Grounded 3D Visual Representations for View-Generalizable Robotic Manipulation

TL;DR

GEM3D presents a unified framework that learns geometrically grounded 3D representations from single-view RGB-D data and distills that knowledge into a monocular visuomotor policy, enabling strong view-generalization in robotic manipulation. The core is a three-stage pretraining pipeline that builds holistic 3D geometry and appearance via dense volumetric features, seed-point reconstruction, and feed-forward Gaussian splatting for multi-view rendering. A multi-step distillation-based policy then aligns a lightweight policy encoder with the pretrained 3D representation and incorporates latent dynamics for temporal consistency, yielding significant gains on RLBench across 12 tasks and robust zero-shot view generalization under viewpoint shifts. Ablation studies confirm the necessity of each component, including deformable attention, Snowflake-style refinement, focal supervision, and the Gaussian splatting renderer, while multi-task and qualitative analyses underscore the method’s practical impact on reliable, viewpoint-robust robotic manipulation.

Abstract

Real-world robotic manipulation demands visuomotor policies capable of robust spatial scene understanding and strong generalization across diverse camera viewpoints. While recent advances in 3D-aware visual representations have shown promise, they still suffer from several key limitations, including reliance on multi-view observations during inference which is impractical in single-view restricted scenarios, incomplete scene modeling that fails to capture holistic and fine-grained geometric structures essential for precise manipulation, and lack of effective policy training strategies to retain and exploit the acquired 3D knowledge. To address these challenges, we present MethodName, a unified representation-policy learning framework for view-generalizable robotic manipulation. MethodName introduces a single-view 3D pretraining paradigm that leverages point cloud reconstruction and feed-forward gaussian splatting under multi-view supervision to learn holistic geometric representations. During policy learning, MethodName performs multi-step distillation to preserve the pretrained geometric understanding and effectively transfer it to manipulation skills. We conduct experiments on 12 RLBench tasks, where our approach outperforms the previous state-of-the-art method by 12.7% in average success rate. Further evaluation on six representative tasks demonstrates strong zero-shot view generalization, with success rate drops of only 22.0% and 29.7% under moderate and large viewpoint shifts respectively, whereas the state-of-the-art method suffers larger decreases of 41.6% and 51.5%.

Figures (13)

• Figure 1: We present GEM3D, a unified rerepresentation-policy learning framework for view-generalizable robotic manipulation.
• Figure 1: Volumetric observation of the Single-view input $o^t$. The left is the 3D Occupancy Map of size $(d^3, 10)$, and the right is the 3D Feature Map of size $(d^3, 384)$.
• Figure 2: Overview of GEM3D.GEM3D comprises two key components: (1) GEM3D Pretraining, which learns holistic 3D representations through auxiliary scene reconstruction tasks; and (2) GEM3D Policy, which distills the pretrained 3D visual representations into a visuomotor policy for view-generalizable manipulation.
• Figure 2: RLBench tasks. 12 tasks of 9 scenes are selected.
• Figure 3: GEM3D Pretraining Pipeline.(a) encoding single-view RGB-D observations into volumetric features, (b) progressively reconstructing scene geometry in a coarse-to-fine Snowflake xiang2021snowflakenet manner, and (c) learning fine-grained texture details through Gaussian-splatting-based novel view rendering.