Deep Whole-body Parkour

TL;DR

Deep Whole-body Parkour addresses the challenge of enabling humanoid robots to perform multi-contact interactions in unstructured environments by unifying exteroceptive perception with whole-body motion tracking. The method learns a single end-to-end policy that adapts reference parkour motions based on depth occupancy maps, using a large-scale simulation pipeline with adaptive sampling and a relative-frame reward scheme to bridge the sim-to-real gap. Key contributions include a depth-annotated dataset, retargeting to a real humanoid, a massively parallel training framework with a custom grouped ray-caster, and real-world deployment showing robust vaulting, kneel climbs, and dive rolls on varied terrains. The work advances toward scalable, general-purpose humanoid controllers capable of contact-rich behavior in real-world settings.

Abstract

Current approaches to humanoid control generally fall into two paradigms: perceptive locomotion, which handles terrain well but is limited to pedal gaits, and general motion tracking, which reproduces complex skills but ignores environmental capabilities. This work unites these paradigms to achieve perceptive general motion control. We present a framework where exteroceptive sensing is integrated into whole-body motion tracking, permitting a humanoid to perform highly dynamic, non-locomotion tasks on uneven terrain. By training a single policy to perform multiple distinct motions across varied terrestrial features, we demonstrate the non-trivial benefit of integrating perception into the control loop. Our results show that this framework enables robust, highly dynamic multi-contact motions, such as vaulting and dive-rolling, on unstructured terrain, significantly expanding the robot's traversability beyond simple walking or running. https://project-instinct.github.io/deep-whole-body-parkour

Figures (7)

• Figure 1: Data-driven whole-body control framework. Real-world environment scans and human demonstrations are processed and aligned to generate feasible motion-terrain pairs. A policy is trained via large-scale reinforcement learning with exteroceptive observations, enabling the robot to replicate agile behaviors in the real world.
• Figure 2: We illustrate the basic concept of relative frame. It has the same x-y and yaw coordinates as the robot base frame and has the same z, roll and pitch coordinate in the reference frame.
• Figure 3: To bridge the sim-to-real gap in depth visualization, we apply several noise patterns in simulation and applied inpainting algorithm from GPU-based OpenCV implementation.
• Figure 4: We show the $x-y$ position variance in a single batch of motion reference example to illustrate the emergence of positional correction ability when introducing depth vision to the end-to-end motion tracking system.
• Figure 5: We do a grid search around the starting position of the motion reference frame. We plot the headmap of the success rate of each motion at a $1.2m \times 1.2m$ space. Red suggests the $100\%$ success rate, while dark blue suggests the $0\%$ success rate.