Hyperbolic Multiview Pretraining for Robotic Manipulation

TL;DR

The proposed HyperMVP, a self-supervised framework for 3D-aware pretraining in a non-Euclidean space for learning robust and generalizable robotic manipulation policies, consistently outperforms strong baselines across diverse tasks and perturbation settings.

Abstract

3D-aware visual pretraining has proven effective in improving the performance of downstream robotic manipulation tasks. However, existing methods are constrained to Euclidean embedding spaces, whose flat geometry limits their ability to model structural relations among embeddings. As a result, they struggle to learn structured embeddings that are essential for robust spatial perception in robotic applications. To this end, we propose HyperMVP, a self-supervised framework for \underline{Hyper}bolic \underline{M}ulti\underline{V}iew \underline{P}retraining. Hyperbolic space offers geometric properties well suited for capturing structural relations. Methodologically, we extend the masked autoencoder paradigm and design a GeoLink encoder to learn multiview hyperbolic representations. The pretrained encoder is then finetuned with visuomotor policies on manipulation tasks. In addition, we introduce 3D-MOV, a large-scale dataset comprising multiple types of 3D point clouds to support pretraining. We evaluate HyperMVP on COLOSSEUM, RLBench, and real-world scenarios, where it consistently outperforms strong baselines across diverse tasks and perturbation settings. Our results highlight the potential of 3D-aware pretraining in a non-Euclidean space for learning robust and generalizable robotic manipulation policies.

Figures (3)

• Figure 1: Overview of HyperMVP. (a) Illustration of the HyperMVP framework, including the 3D-MOV pretraining dataset, embedding spaces, and downstream applications. (b) Comparison of generalization performance (%) on the Colosseum pumacay2024colosseum under various perturbation settings. Results of HyperMVP are averaged over three evaluation runs, while other methods follow the single-run reports in qian20253dmvppumacay2024colosseum
• Figure 2: Pipeline of HyperMVP. During pretraining, a GeoLink encoder is pretrained on multiview images rendered from point clouds. During finetuning, the pretrained GeoLink encoder is finetuned with the Robotic View Transformer for manipulation tasks.
• Figure 3: Qualitative reconstruction results. left: scene-level multiview inputs for validating effectiveness. middle: the unseen object within seen categories (instance-level generalization). right: the object from unseen categories (domain-level generalization).