Lift3D Foundation Policy: Lifting 2D Large-Scale Pretrained Models for Robust 3D Robotic Manipulation

TL;DR

Lift3D tackles the challenge of robust 3D robotic manipulation with limited 3D data by two-stagely enriching 2D foundation models: first improving implicit 3D understanding via a task-aware MAE with depth reconstruction, then enabling explicit 3D reasoning by mapping 3D point clouds to the 2D backbone's positional embeddings using virtual planes and a 3D tokenizer. This approach preserves large-scale pretrained knowledge while minimizing spatial information loss, yielding strong results across simulation suites and real-world tasks and demonstrating notable scalability with larger 2D foundations. Key contributions include the task-aware MAE design, the 2D model-lifting strategy for point clouds, and extensive ablation and generalization analyses. The method promises practical impact by enabling robust 3D manipulation with efficient use of existing 2D pretrained resources, though language-conditioned capabilities remain for future work.

Abstract

3D geometric information is essential for manipulation tasks, as robots need to perceive the 3D environment, reason about spatial relationships, and interact with intricate spatial configurations. Recent research has increasingly focused on the explicit extraction of 3D features, while still facing challenges such as the lack of large-scale robotic 3D data and the potential loss of spatial geometry. To address these limitations, we propose the Lift3D framework, which progressively enhances 2D foundation models with implicit and explicit 3D robotic representations to construct a robust 3D manipulation policy. Specifically, we first design a task-aware masked autoencoder that masks task-relevant affordance patches and reconstructs depth information, enhancing the 2D foundation model's implicit 3D robotic representation. After self-supervised fine-tuning, we introduce a 2D model-lifting strategy that establishes a positional mapping between the input 3D points and the positional embeddings of the 2D model. Based on the mapping, Lift3D utilizes the 2D foundation model to directly encode point cloud data, leveraging large-scale pretrained knowledge to construct explicit 3D robotic representations while minimizing spatial information loss. In experiments, Lift3D consistently outperforms previous state-of-the-art methods across several simulation benchmarks and real-world scenarios.

Figures (8)

• Figure 1: Lift3D empowers 2D foundation models with 3D manipulation capabilities by refining implicit 3D robotic representations through task-related affordance masking and depth reconstruction, while enhancing explicit 3D robotic representations by leveraging the pretrained 2D positional embeddings to encode point cloud. Lift3D achieves robustness and surprising effectiveness in diverse simulation and real-world tasks.
• Figure 2: Overall pipeline of Lift3D.a) For implicit 3D robotic representation, we leverage CLIP radford2021learning to offline extract image attention maps based on task descriptions, which are back-projected onto the 2D input to guide the MAE masking. We then input the visible tokens into the 2D foundation model to extract features. The masked tokens and encoded visible tokens are processed by the MAE decoder for depth reconstruction, enhancing 3D spatial awareness. Meanwhile, the encoded visible tokens are also distilled using corresponding features from the off-the-shelf pretrained model to mitigate catastrophic forgetting. b) For explicit 3D robotic representation, we first project the point cloud data onto multiple virtual planes, establishing a positional mapping between the 3D input points and the 2D positional embeddings (PEs) on each virtual plane. After mapping, we average the 2D PEs corresponding to each 3D patch to form a unified positional indicator(3D PEs), which is then integrated with the 3D tokens. These 3D tokens are generated by feeding the point cloud into a lightweight 3D tokenizer. Finally, the output features from the 2D foundation model are processed through a policy head to predict the pose for imitation learning.
• Figure 3: Quantitative results for real robot experiments, where the y-axis represents the manipulation success rate.
• Figure 4: The qualitative results of Lift3D in real-world experiments, including the input point cloud examples, manipulation progress, and the task completion end state, are shown. More visualizations can be found in Appendix \ref{['apsec: AQE2']}.
• Figure 5: Scalability. Y-axis is the manipulation success rate.