CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining

TL;DR

CLAMP tackles the need for 3D-grounded pretraining in robotic manipulation by jointly learning image, text, and action representations from 3D multi-view observations and action histories. It introduces STRING-based relative positional encoding within a Vision Transformer to fuse five fixed/dynamic views, and a text encoder grounded in spatial-task information, all trained with cross-modal contrastive losses alongside a diffusion-based visuomotor policy pretraining. The method yields substantial gains in sample efficiency and final performance on both simulated and real tasks, outperforming strong baselines and ablations. By pretraining encoders and the policy in tandem and leveraging 3D perception, CLAMP advances practical transfer to unseen tasks in robotics. This approach highlights the value of 3D data and action-conditioned signals for robust, data-efficient robot manipulation.

Abstract

Leveraging pre-trained 2D image representations in behavior cloning policies has achieved great success and has become a standard approach for robotic manipulation. However, such representations fail to capture the 3D spatial information about objects and scenes that is essential for precise manipulation. In this work, we introduce Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining (CLAMP), a novel 3D pre-training framework that utilizes point clouds and robot actions. From the merged point cloud computed from RGB-D images and camera extrinsics, we re-render multi-view four-channel image observations with depth and 3D coordinates, including dynamic wrist views, to provide clearer views of target objects for high-precision manipulation tasks. The pre-trained encoders learn to associate the 3D geometric and positional information of objects with robot action patterns via contrastive learning on large-scale simulated robot trajectories. During encoder pre-training, we pre-train a Diffusion Policy to initialize the policy weights for fine-tuning, which is essential for improving fine-tuning sample efficiency and performance. After pre-training, we fine-tune the policy on a limited amount of task demonstrations using the learned image and action representations. We demonstrate that this pre-training and fine-tuning design substantially improves learning efficiency and policy performance on unseen tasks. Furthermore, we show that CLAMP outperforms state-of-the-art baselines across six simulated tasks and five real-world tasks.

Figures (9)

• Figure 1: CLAMP is a 3D pre-training framework for robotic manipulation. We contrastively pre-train an image encoder with fixed and dynamic multi-view observations, an action encoder with histories of action chunks, and a text encoder with spatial and temporal object- and task-level information. CLAMP learns image and action representations that improve downstream policy fine-tuning efficiency and performance. The bottom figure compares the ALOHA Unleashed evaluation curves during fine-tuning without and with CLAMP on the Mug on Plate task.
• Figure 2: Overview of CLAMP. (i): CLAMP consists of three encoders: image, action and text. The image encoder is a Vision Transformer (ViT) dosovitskiy2020image that takes five multi-view observations (overhead, back-right, front-left, wrist-left, and wrist-right) as input. These views are rendered from a merged point cloud and include depth and 3D coordinates. The action encoder is a Transformer encoder that takes a history of previous actions ($a_{t-1},\dots,a_{t-H}$) as input. The text input includes a general task description, each object's name and normalized position in the global coordinate frame, and an integer indicating task progress, and is processed by a CLIP radford2021learning text encoder. The intuition is to learn image and action representations that capture correlations between action patterns and robot states as observed in images, grounded by text that describes the spatial and temporal information about the objects and task. We also pre-train a Diffusion Policy zhao2024aloha in parallel. (ii): The Diffusion Policy is first initialized with the pre-trained weights from stage (i) and use the frozen CLAMP encoders to extract image and action embeddings. These embeddings are fed into a Transformer encoder together with RGB feature maps from ResNet-50 backbones and proprioceptive features from the robot to predict noise of shape $50 \times 14$ for the next 50 actions ($\epsilon_{t+1},\dots, \epsilon_{t+50}$).
• Figure 3: Pictorial desciption of the STRING mechanism applied in CLAMP's image encoder to correlate tokens form different views corresponding to similar regions of the 3D space. Tokens from different 2D views are concatenated, but also equipped with STRING positional encoding mechanism schenck2025learning. Consequently, the attention score between two highlighted tokens from two different views will be modulated by a function that depends only on the vector $\mathbf{r}_{i,j} \in \mathbb{R}^{3}$ between the centers of mass of two sub-point clouds (highlighted in red and blue correspondingly) of the visible sets of points related to those tokens.
• Figure 4: Evaluation curves comparing our method (green) to baselines across three seeds in simulation. All methods are trained on 4500 demonstrations for 1M environment steps and evaluated for 50 trials per checkpoint every 40K steps.
• Figure 5: Successful rollouts of our method on the ALOHA robot. Top: Mug on Plate. Bottom: Open Drawers.

Theorems & Definitions (4)

• Lemma 3.1
• Lemma 3.2
• proof
• proof