Videos are Sample-Efficient Supervisions: Behavior Cloning from Videos via Latent Representations

TL;DR

BCV-LR tackles the challenge of learning policies from action-free videos with no rewards by introducing offline latent pretraining to extract action-related features and latent actions, followed by online reward-free finetuning to align these latent actions with real actions for behavior cloning. The method combines a self-supervised encoder, a discretized latent-action predictor, and a world model to predict latent transitions, then uses a decoder and latent policy to realize actions in the environment, resulting in iterative, highly data-efficient policy improvement. Experiments across Procgen, DMControl, and Metaworld show BCV-LR achieving expert-level or near-expert performance with limited interactions, surpassing state-of-the-art ILV and reward-based baselines in data efficiency. This work demonstrates that videos can serve as a powerful, scalable supervision signal for visual policy learning, with implications for real-world robotics where explicit rewards or expert actions are scarce.

Abstract

Humans can efficiently extract knowledge and learn skills from the videos within only a few trials and errors. However, it poses a big challenge to replicate this learning process for autonomous agents, due to the complexity of visual input, the absence of action or reward signals, and the limitations of interaction steps. In this paper, we propose a novel, unsupervised, and sample-efficient framework to achieve imitation learning from videos (ILV), named Behavior Cloning from Videos via Latent Representations (BCV-LR). BCV-LR extracts action-related latent features from high-dimensional video inputs through self-supervised tasks, and then leverages a dynamics-based unsupervised objective to predict latent actions between consecutive frames. The pre-trained latent actions are fine-tuned and efficiently aligned to the real action space online (with collected interactions) for policy behavior cloning. The cloned policy in turn enriches the agent experience for further latent action finetuning, resulting in an iterative policy improvement that is highly sample-efficient. We conduct extensive experiments on a set of challenging visual tasks, including both discrete control and continuous control. BCV-LR enables effective (even expert-level on some tasks) policy performance with only a few interactions, surpassing state-of-the-art ILV baselines and reinforcement learning methods (provided with environmental rewards) in terms of sample efficiency across 24/28 tasks. To the best of our knowledge, this work for the first time demonstrates that videos can support extremely sample-efficient visual policy learning, without the need to access any other expert supervision.

Figures (10)

• Figure 1: BCV-LR achieves sample-efficient video-based imitation learning without accessing expert actions or rewards. It achieves expert-level policy performance on discrete task "Bossfight" and continuous task "reacher_hard" with only 100k interactions allowed, surpassing state-of-the-art ILV and RL baselines.
• Figure 2: The training objectives of different stages. BCV-LR first pre-trains a self-supervised feature encoder $f$ over the videos. Based on the latent features, BCV-LR employs another trainable world model $w$ along with the latent action predictor $p$, optimizing a dynamics-based objective in an unsupervised manner to obtain the latent actions between consecutive video frames. In the online stage, BCV-LR fine-tunes the latent actions with the pretrained world model $w$ over the collected reward-free transitions, aligning latent actions to the real action space via a latent action decoder $d$. Simultaneously, BCV-LR trains a latent policy $\pi$ that clones the latent actions, which shares the latent feature encoder $f$ and latent action decoder $d$ to interact with the environment. This enriches collected data for further latent action finetuning, resulting in an iterative improvement. Note that $f$, $\pi$, and $d$ together form the final policy of BCV-LR.
• Figure 3: Online training curves of ILV methods in DMControl. BCV-LR can efficiently utilize environmental samples and learn effective strategies at 50k steps (even 20k on some tasks).
• Figure 4: Ablation study on both discrete control and continuous control.
• Figure 5: The training curves of BCV-LR when given different numbers of action-free video transitions. 50k video transitions are enough for BCV-LR to learn an effective policy.