Reinforcement Learning from Wild Animal Videos

TL;DR

RLWAV addresses how to learn legged locomotion by leveraging large-scale wild-animal videos as a source of natural motion, bypassing the need for reference trajectories. It trains a video action classifier on the Animal Kingdom dataset to recognize four behaviors and uses the classifier score as a reward signal to train a multi-skill policy in a physics simulator under constrained RL, then transfers the learned behaviors to a real Solo-12 quadruped. The approach demonstrates the emergence of keeping still, walking, and jumping across a substantial cross-embodiment gap, without pose tracking or per-skill rewards. While promising, the work notes limitations relative to state-of-the-art locomotion and suggests future work in more targeted video datasets and advanced video understanding to further improve agility and stability.

Abstract

We propose to learn legged robot locomotion skills by watching thousands of wild animal videos from the internet, such as those featured in nature documentaries. Indeed, such videos offer a rich and diverse collection of plausible motion examples, which could inform how robots should move. To achieve this, we introduce Reinforcement Learning from Wild Animal Videos (RLWAV), a method to ground these motions into physical robots. We first train a video classifier on a large-scale animal video dataset to recognize actions from RGB clips of animals in their natural habitats. We then train a multi-skill policy to control a robot in a physics simulator, using the classification score of a third-person camera capturing videos of the robot's movements as a reward for reinforcement learning. Finally, we directly transfer the learned policy to a real quadruped Solo. Remarkably, despite the extreme gap in both domain and embodiment between animals in the wild and robots, our approach enables the policy to learn diverse skills such as walking, jumping, and keeping still, without relying on reference trajectories nor skill-specific rewards.

Figures (7)

• Figure 1: RLWAV trains a video classifier on 8,791 wild animal videos in natural environments to transfer the learned skills to a real quadruped robot with reinforcement learning, without relying on reference trajectories nor hand-designed rewards for each skill. Here, we present the motions resulting from the transfer of the keeping still, walking, and jumping skills to the Solo-12 robot.
• Figure 2: (Left) We train a video classifier to recognize actions from the Animal Kingdom dataset ng2022animal. (Right) With a third-person camera capturing videos of the robot’s movement, we use the classification score for the desired skill as a reward to train the policy with Constrained RL.
• Figure 3: Impact of animal video dataset size (% of total videos) on skill emergence.
• Figure 4: (Top) Example of failure case for the ”Walking” skill, where the robot performs walking motions for only two legs without moving forward, fooling the video reward function. (Bottom) Base camera in nominal setting, camera 1,2,3,4 in the ablation study.
• Figure 5: Rollout examples for the 4 skills considered on the real Solo-12.