Do's and Don'ts: Learning Desirable Skills with Instruction Videos

TL;DR

DoDont addresses unsafe behaviors in unsupervised skill discovery by learning a reward-shaping signal from action-free instruction videos and using an instruction network as the distance metric within a distance-maximizing framework. It demonstrates data-efficient learning of complex locomotion and manipulation skills with as few as eight videos, while consistently avoiding undesirable actions across challenging environments. The method outperforms baselines like METRA, SMERL, and DGPO, particularly in high-dimensional and pixel-based tasks, and extends to offline zero-shot RL. By injecting human intention through inexpensive video data, DoDont offers a practical, scalable approach to safer and more capable USD in real-world robotics and control settings.

Abstract

Unsupervised skill discovery is a learning paradigm that aims to acquire diverse behaviors without explicit rewards. However, it faces challenges in learning complex behaviors and often leads to learning unsafe or undesirable behaviors. For instance, in various continuous control tasks, current unsupervised skill discovery methods succeed in learning basic locomotions like standing but struggle with learning more complex movements such as walking and running. Moreover, they may acquire unsafe behaviors like tripping and rolling or navigate to undesirable locations such as pitfalls or hazardous areas. In response, we present DoDont (Do's and Don'ts), an instruction-based skill discovery algorithm composed of two stages. First, in an instruction learning stage, DoDont leverages action-free instruction videos to train an instruction network to distinguish desirable transitions from undesirable ones. Then, in the skill learning stage, the instruction network adjusts the reward function of the skill discovery algorithm to weight the desired behaviors. Specifically, we integrate the instruction network into a distance-maximizing skill discovery algorithm, where the instruction network serves as the distance function. Empirically, with less than 8 instruction videos, DoDont effectively learns desirable behaviors and avoids undesirable ones across complex continuous control tasks. Code and videos are available at https://mynsng.github.io/dodont/

Figures (10)

• Figure 1: (a) The offline instruction video dataset includes videos of desirable behaviors (Do's) and undesirable behaviors (Don'ts). (b) Unsupervised skill discovery algorithms tend to learn undesirable behaviors. (c) In DoDont, an instruction network is first trained with the Do's and Don'ts videos to distinguish desirable and undesirable behaviors. Then, this instruction network adjusts the intrinsic reward of the skill discovery algorithm, promoting desirable skills while avoiding undesirable ones.
• Figure 2: Benchmark environments.
• Figure 3: Left: State coverage and zero-shot task reward for Cheetah and Quadruped. Right: Visualization of Do videos in our instruction video dataset and learned skills by DoDont. We are able to observe that DoDont does not simply mimic instruction videos but extracts desirable behaviors (e.g., run) from the videos and learn diverse skills.
• Figure 4: Visualization and comparison of learned skills. In both environments, the left side is hazardous and the right side is safe. Safe state coverage assesses the agent's ability to cover safe areas and avoid hazards.
• Figure 5: Learning safe and diverse behaviors. Zero-shot rewards assess how effectively each method learns desired behaviors while avoiding hazardous ones.