CLAP: Contrastive Latent Action Pretraining for Learning Vision-Language-Action Models from Human Videos

TL;DR

This work tackles the data scarcity challenge in robotic manipulation by introducing Contrastive Latent Action Pretraining (CLAP), which explicitly aligns a visual latent space from unlabeled human videos with an executable robot latent-action space learned from trajectories via Act-VAE and VD-VAE. Building on this alignment, it presents a dual-formulation Vision-Language-Action framework: CLAP-NTP for discrete, instruction-following planning and CLAP-RF for high-frequency, precise control through Rectified Flow, augmented with Knowledge Matching to prevent catastrophic forgetting during fine-tuning. The approach achieves state-of-the-art results on real-world bimanual tasks and LIBERO simulation, demonstrating robust object generalization, resilience to perceptual perturbations, and efficient inference. Collectively, CLAP enables effective transfer of human-video priors to robotic execution, bridging semantic understanding and motor control across embodiments with practical impact for generalist robotics.

Abstract

Generalist Vision-Language-Action models are currently hindered by the scarcity of robotic data compared to the abundance of human video demonstrations. Existing Latent Action Models attempt to leverage video data but often suffer from visual entanglement, capturing noise rather than manipulation skills. To address this, we propose Contrastive Latent Action Pretraining (CLAP), a framework that aligns the visual latent space from videos with a proprioceptive latent space from robot trajectories. By employing contrastive learning, CLAP maps video transitions onto a quantized, physically executable codebook. Building on this representation, we introduce a dual-formulation VLA framework offering both CLAP-NTP, an autoregressive model excelling at instruction following and object generalization, and CLAP-RF, a Rectified Flow-based policy designed for high-frequency, precise manipulation. Furthermore, we propose a Knowledge Matching (KM) regularization strategy to mitigate catastrophic forgetting during fine-tuning. Extensive experiments demonstrate that CLAP significantly outperforms strong baselines, enabling the effective transfer of skills from human videos to robotic execution. Project page: https://lin-shan.com/CLAP/.

Figures (9)

• Figure 1: Visualization of our aligned latent action space. We display samples from clustered action tokens, demonstrating semantic alignment across diverse robots (Astribot, AgiBot) and human (Ego4D) domains. Groups 1–3 correspond to moving right, placing, and grasping, respectively. The red arrows on the Astribot S1 frames visualize the predicted 3D trajectory decoded from the latent action and projected onto the image plane, confirming the physical executability of the learned representations.
• Figure 2: Overview of CLAP. Unlike (a) conventional methods that rely solely on limited robot teleoperation data, (b) CLAP learns an executable latent action space from large-scale human demonstrations. This enables the transfer of semantic knowledge to robot policies, achieving objects generalization through human videos.
• Figure 3: The pipeline of CLAP. (a) Contrastive Latent Action Pretraining: Visual state transitions from videos are aligned with quantized robot actions via contrastive learning to establish a shared, physically grounded latent space. (b) VLA Frameworks: We introduce CLAP-NTP for discrete autoregressive planning and CLAP-RF for continuous high-frequency control via a Rectified Flow expert.
• Figure 4: Knowledge matching algorithm.Grey blocks represent the input observations and instructions. Blue blocks denote the subtask and discrete action tokens, where $\mathcal{L}_{\text{KL}}$ constrains the policy distribution. Green blocks represent the continuous actions, trained via $\mathcal{L}_{\text{RF}}$.
• Figure 5: The experiment setup. The Robot Configuration (top) features the Astribot S1 with dual 7-DoF arms and a multi-camera perception suite. VR Teleoperation (bottom) is performed using a Meta Quest 3S headset to collect human demonstration data.