Sample-efficient Unsupervised Policy Cloning from Ensemble Self-supervised Labeled Videos

TL;DR

UPESV tackles the problem of learning effective policies with minimal supervision by using action-free videos and reward-free interactions. It introduces three self-supervised tasks—Visual Shift Contrast, Latent Future Reconstruction, and Ground-truth Action Prediction—to train a robust video labeling model that infers expert actions and guides policy cloning. The method jointly optimizes labeling and policy, and experiments across sixteen Procgen tasks show state-of-the-art performance under limited interactions, outperforming several baselines. The approach reduces reliance on rewards and labeled trajectories, with practical implications for learning from easily accessible video data, albeit with higher computational cost.

Abstract

Current advanced policy learning methodologies have demonstrated the ability to develop expert-level strategies when provided enough information. However, their requirements, including task-specific rewards, action-labeled expert trajectories, and huge environmental interactions, can be expensive or even unavailable in many scenarios. In contrast, humans can efficiently acquire skills within a few trials and errors by imitating easily accessible internet videos, in the absence of any other supervision. In this paper, we try to let machines replicate this efficient watching-and-learning process through Unsupervised Policy from Ensemble Self-supervised labeled Videos (UPESV), a novel framework to efficiently learn policies from action-free videos without rewards and any other expert supervision. UPESV trains a video labeling model to infer the expert actions in expert videos through several organically combined self-supervised tasks. Each task performs its duties, and they together enable the model to make full use of both action-free videos and reward-free interactions for robust dynamics understanding and advanced action prediction. Simultaneously, UPESV clones a policy from the labeled expert videos, in turn collecting environmental interactions for self-supervised tasks. After a sample-efficient, unsupervised, and iterative training process, UPESV obtains an advanced policy based on a robust video labeling model. Extensive experiments in sixteen challenging procedurally generated environments demonstrate that the proposed UPESV achieves state-of-the-art interaction-limited policy learning performance (outperforming five current advanced baselines on 12/16 tasks) without exposure to any other supervision except for videos.

Figures (4)

• Figure 1: UPESV achieves sample-efficient policy learning with only action-free videos and reward-free interactions. This is achieved by three organically combined self-supervised tasks, where each performs its duties. For example, on the interaction-limited Fruitbot task, UPESV obtains the only effective policy, where each self-supervised task is necessary.
• Figure 2: UPESV framework. UPESV learns a video labeling model and a policy network jointly through three organically combined self-supervised tasks, where each is necessary and performs its own duties. The motivation and details of these three tasks (visual shift contrast, latent future reconstruction, and ground-truth action prediction) are provided separately in Sections III.A, III.B, and III.C. Simultaneously, we imitate a policy $\pi(a|o)$ by behavior cloning the labeled expert videos. The policy interacts with the reward-free environment, in turn enriching the self-supervised training data for the video labeling model. The labeling model and policy are improved iteratively, which we detail in Section III.D.
• Figure 3: Ablation study of three self-supervised tasks on eight procgen environments. Each task is necessary in the proposed UPESV.
• Figure 4: Hyper-parameter analysis of visual shift contrast.