OGMP: Oracle Guided Multi-mode Policies for Agile and Versatile Robot Control

TL;DR

This work tackles the difficulty of applying deep RL to agile legged robot control by introducing Oracle Guided Multi-mode Policy (OGMP), which combines closed-loop oracle references with bounded exploration to avoid poor local optima. The framework couples Oracle Guided Policy Optimization (OGPO) with Task Vital Multi-modality, enabling a single policy to master a finite set of modes and transitions that generalize to infinite-horizon tasks. A mode encoder maps oracle-generated modal trajectories into a compact latent space, and a mode-conditioned policy is trained with PPO using observations that include proprioception, the latent mode, a clock, and terrain cues, with a surrogate tracking objective and a termination criterion based on proximity to the oracle reference. Experimental results on a 16-DoF biped (HECTOR) show parkour across diverse tracks and 2 m dives in simulation, plus successful sim-to-real transfer for parkour, illustrating versatile agility and robust mode transitions.

Abstract

The efficacy of reinforcement learning for robot control relies on the tailored integration of task-specific priors and heuristics for effective exploration, which challenges their straightforward application to complex tasks and necessitates a unified approach. In this work, we define a general class for priors called oracles that generate state references when queried in a closed-loop manner during training. By bounding the permissible state around the oracle's ansatz, we propose a task-agnostic oracle-guided policy optimization. To enhance modularity, we introduce task-vital modes, showing that a policy mastering a compact set of modes and transitions can handle infinite-horizon tasks. For instance, to perform parkour on an infinitely long track, the policy must learn to jump, leap, pace, and transition between these modes effectively. We validate this approach in challenging bipedal control tasks: parkour and diving using a 16 DoF dynamic bipedal robot, HECTOR. Our method results in a single policy per task, solving parkour across diverse tracks and omnidirectional diving from varied heights up to 2m in simulation, showcasing versatile agility. We demonstrate successful sim-to-real transfer of parkour, including leaping over gaps up to 105 % of the leg length, jumping over blocks up to 20 % of the robot's nominal height, and pacing at speeds of up to 0.6 m/s, along with effective transitions between these modes in the real robot.

Figures (6)

• Figure 1: Overview of OGMP: Oracle guided policy optimization and the applied tasks visualized. Trained OGMPs $\pi_\text{parkour}$ performing agile parkour in the simulation and the real-robot. $\pi_\text{dive}$ performing a frontflip dive from a $2$m high platform in simulation. Accompanying video results at : https://youtu.be/69SVc-43Oqg?si=w4r3i67oBaoThLN7
• Figure 2: Oracle bounded exploration
• Figure 3: Overview of the design methodology: a) The breakdown of a task into its mode and mode parameter set b) Guided exploration by bounded permissible state space around the local neighborhood of the oracle's reference c) Mode encoder: an LSTM autoencoder trained on a custom modal dataset by minimizing reconstruction loss d) multi-mode policy trained with oracle guided policy optimization on a task environment
• Figure 4: Base height reference trajectories from various oracles for different modes
• Figure 5: Keyframes of an OGMP ($\pi_\text{parkour}$) demonstrating mode transitions (left to the right) with the color of the active mode marked: blue for pace, red for leap, and green for jump.