Extreme Parkour with Legged Robots

TL;DR

This work demonstrates that extreme parkour can be learned end-to-end on a low-cost legged robot using a single depth image to drive motor commands. The authors introduce a two-phase training framework with dual distillation and an inner-product reward that enables autonomous heading and diverse maneuvers, achieving 2x the robot’s height in high jumps, 2x its length in long jumps, and a handstand gait, both in simulation and on hardware. Key contributions include the phase-wise ROA-based environment adaptation, the scandots-to-depth distillation approach, and a unified reward design that generalizes across obstacle geometries. The results show robust, depth-only parkour capabilities and strong real-world performance, highlighting the potential for deploying agile locomotion on affordable platforms without explicit maps or planning.

Abstract

Humans can perform parkour by traversing obstacles in a highly dynamic fashion requiring precise eye-muscle coordination and movement. Getting robots to do the same task requires overcoming similar challenges. Classically, this is done by independently engineering perception, actuation, and control systems to very low tolerances. This restricts them to tightly controlled settings such as a predetermined obstacle course in labs. In contrast, humans are able to learn parkour through practice without significantly changing their underlying biology. In this paper, we take a similar approach to developing robot parkour on a small low-cost robot with imprecise actuation and a single front-facing depth camera for perception which is low-frequency, jittery, and prone to artifacts. We show how a single neural net policy operating directly from a camera image, trained in simulation with large-scale RL, can overcome imprecise sensing and actuation to output highly precise control behavior end-to-end. We show our robot can perform a high jump on obstacles 2x its height, long jump across gaps 2x its length, do a handstand and run across tilted ramps, and generalize to novel obstacle courses with different physical properties. Parkour videos at https://extreme-parkour.github.io/

Figures (7)

• Figure 1: Extreme Parkour: Low-cost robot with imprecise actuation can perform precise athletic behaviors directly from a high-dimensional image without any explicit mapping and planning. The robot is able to long jump across gaps $2\times$ of its own length, high jump over obstacles $2\times$ its own height, run over tilted ramps, and walk on just front two legs (handstand) -- all with a single neural network operating directly on depth from a single, front-facing camera. Parkour videos at https://extreme-parkour.github.io/.
• Figure 2: Training overview. In phase 1, we use RL to learn a locomotion policy with access to privileged information like environment parameters and scandots agarwal2022legged in addition to heading direction from waypoints. We use Regularized Online Adaptation (ROA)fu2021minimizing to train an estimator to recover environmental information from the history of observations. In phase 2, we distill from scandots into a policy that operates from onboard depth and automatically decides its heading (yaw) direction conditioned on the obstacle.
• Figure 3: Terrains in simulation with red dots indicating waypoints that are used to get heading direction.
• Figure 4: Key frames of our robot executing a very high jump (2x its height). We note the emergent foot placement, power generated through hind legs and climbing behavior from the front legs.
• Figure 5: Keyframes from a long jump (2x robot length)