CoMo: Learning Continuous Latent Motion from Internet Videos for Scalable Robot Learning

TL;DR

CoMo introduces a continuous latent-motion representation learned from action-less internet videos using an early temporal feature difference, mitigating shortcut learning and enabling scalable, unified policy training with pseudo actions. Guided by the information bottleneck, CoMo fixes latent dimensionality to balance motion detail with noise, and it provides LP-MSE and S-PCFC metrics for robust, low-cost evaluation. The framework supports joint learning with both diffusion-based and autoregressive policies, achieving strong zero-shot generalization to unseen domains and improving real-world robot performance when trained with pseudo actions. Overall, CoMo demonstrates scalable cross-domain robot learning by leveraging diverse, action-less video data to augment traditional robot datasets. Its practical impact lies in reducing data requirements while achieving superior policy performance across simulated and real-world tasks.

Abstract

Learning latent motion from Internet videos is crucial for building generalist robots. However, existing discrete latent action methods suffer from information loss and struggle with complex and fine-grained dynamics. We propose CoMo, which aims to learn more informative continuous motion representations from diverse, internet-scale videos. CoMo employs a early temporal feature difference mechanism to prevent model collapse and suppress static appearance noise, effectively discouraging shortcut learning problem. Furthermore, guided by the information bottleneck principle, we constrain the latent motion embedding dimensionality to achieve a better balance between retaining sufficient action-relevant information and minimizing the inclusion of action-irrelevant appearance noise. Additionally, we also introduce two new metrics for more robustly and affordably evaluating motion and guiding motion learning methods development: (i) the linear probing MSE of action prediction, and (ii) the cosine similarity between past-to-current and future-to-current motion embeddings. Critically, CoMo exhibits strong zero-shot generalization, enabling it to generate continuous pseudo actions for previously unseen video domains. This capability facilitates unified policy joint learning using pseudo actions derived from various action-less video datasets (such as cross-embodiment videos and, notably, human demonstration videos), potentially augmented with limited labeled robot data. Extensive experiments show that policies co-trained with CoMo pseudo actions achieve superior performance with both diffusion and autoregressive architectures in simulated and real-world settings.

Figures (9)

• Figure 1: The CoMo177, 31, 19231, 85, 192 framework. In the first stage, we self-supervisely learn inter-frame latent motion representations from Internet videos. In the second stage, we directly utilize the IDM trained in the first stage to extract pseudo action labels for action-less video data, ensuring joint learning of continuous robot action data and action-less video data under a unified policy architecture.
• Figure 2: The Metrics for evaluating the quality of latent motion representations. Better latent motion representations typically lead to higher policy success rate and lower LP-MSE and S-PCFC.
• Figure 3: The latent motion embedding dimension scaling line chart.(a) Our CoMo177, 31, 19231, 85, 192 achieves the highest success rate when the motion dimension is 128 (the better balance between LP-MSE and S-PCFC). (b) Our CoMo177, 31, 19231, 85, 192 obtains a lower S-PCFC as the motion dimension decreases, whereas simply removing vector quantization does not (with severe shortcut learning problem).
• Figure 4: The LIBERO experiment results of CoMo177, 31, 19231, 85, 192 trained on larger-scale Internet videos without any fine-tuning.
• Figure 4: The Latent Motion. We report the cosine similarity of latent motion from action-less human videos, comparing time-reversed actions in the same environment (left) and consistent actions in different environments (right). The latter shows higher cosine similarity values than the former.