ConLA: Contrastive Latent Action Learning from Human Videos for Robotic Manipulation

TL;DR

ConLA tackles the scalability gap between robot-only pretraining and the abundance of human demonstration videos by introducing a contrastive disentanglement framework that separates motion from visual noise in latent actions. It relies on action-category priors and temporal cues to obtain semantically meaningful latent actions from human videos, then pretrains a large vision-language model to predict discrete latent-action tokens before finetuning on a small set of real robot trajectories. Empirically, ConLA achieves state-of-the-art performance across SimplerEnv and real-world tabletop tasks, with human-video pretraining surpassing robot-trajectory pretraining by 1.1% on SimplerEnv and yielding up to 15.9% gains in the real world. This work demonstrates the potential of large-scale human-video pretraining for vision-language robotic control, reducing reliance on expensive robot data while maintaining transferability and motion semantics.

Abstract

Vision-Language-Action (VLA) models achieve preliminary generalization through pretraining on large scale robot teleoperation datasets. However, acquiring datasets that comprehensively cover diverse tasks and environments is extremely costly and difficult to scale. In contrast, human demonstration videos offer a rich and scalable source of diverse scenes and manipulation behaviors, yet their lack of explicit action supervision hinders direct utilization. Prior work leverages VQ-VAE based frameworks to learn latent actions from human videos in an unsupervised manner. Nevertheless, since the training objective primarily focuses on reconstructing visual appearances rather than capturing inter-frame dynamics, the learned representations tend to rely on spurious visual cues, leading to shortcut learning and entangled latent representations that hinder transferability. To address this, we propose ConLA, an unsupervised pretraining framework for learning robotic policies from human videos. ConLA introduces a contrastive disentanglement mechanism that leverages action category priors and temporal cues to isolate motion dynamics from visual content, effectively mitigating shortcut learning. Extensive experiments show that ConLA achieves strong performance across diverse benchmarks. Notably, by pretraining solely on human videos, our method for the first time surpasses the performance obtained with real robot trajectory pretraining, highlighting its ability to extract pure and semantically consistent latent action representations for scalable robot learning.

Figures (8)

• Figure 1: The overview of ConLA, which leverages contrastive learning to disentangle latent actions from visual noise in videos, guiding the construction of compact latent action representations. This enables the model to learn motion priors from complex human videos, improving downstream robot manipulation tasks.
• Figure 2: Illustration of shortcut learning: using the latent action extracted from the first-row frame pair to reconstruct the second-row $O_{t+k}$ fails, as the reconstruction drives the model to capture appearance rather than motion.
• Figure 3: Contrastive Latent Action Learning. We propose a contrastive disentanglement framework to separate action from visual interference in video clips spanning the current and future frames. Specifically, samples with action class labels and their inversely augmented counterparts are encoded into latent action embeddings, which are evenly divided and fed into the Action head for Action-Centric Contrastive Learning and the Visual head for Vision-Centric Contrastive Learning to achieve disentangled representations. The optimized representation from the Action head is further quantized, and the resulting quantized latent actions, together with the current frame $O_t$, are employed to reconstruct the future frame $O_{t+k}$.
• Figure 4: Real-world Manipulation Robot Results. ConLA outperforms prior state-of-the-art by 15.9% in success rate.
• Figure 5: Latent action analysis. Visualization of shortcut learning in latent action extraction. Reconstructed images conditioned on the extracted latent actions demonstrate that our method captures motion-relevant actions, alleviating shortcut learning.