APEX: Action Priors Enable Efficient Exploration for Robust Motion Tracking on Legged Robots

TL;DR

The paper addresses the challenge of deploying motion-tracking policies for legged robots without dependency on demonstration data at runtime and with reduced reward-tuning requirements. It introduces APEX, which augments RL with decaying action priors that bias early exploration toward expert demonstrations and gradually fade, plus a dual-critic PPO framework to separately optimize style and task rewards. A single policy can learn and switch between multiple gaits, and experiments show significant improvements in sample efficiency and robustness, including sim-to-real transfer on a Unitree Go2 across terrains and velocities. This approach reduces engineering overhead while improving stability and generalization, potentially enabling more scalable guidance-driven RL for both locomotion and manipulation tasks.

Abstract

Learning natural, animal-like locomotion from demonstrations has become a core paradigm in legged robotics. Despite the recent advancements in motion tracking, most existing methods demand extensive tuning and rely on reference data during deployment, limiting adaptability. We present APEX (Action Priors enable Efficient Exploration), a plug-and-play extension to state-of-the-art motion tracking algorithms that eliminates any dependence on reference data during deployment, improves sample efficiency, and reduces parameter tuning effort. APEX integrates expert demonstrations directly into reinforcement learning (RL) by incorporating decaying action priors, which initially bias exploration toward expert demonstrations but gradually allow the policy to explore independently. This is combined with a multi-critic framework that balances task performance with motion style. Moreover, APEX enables a single policy to learn diverse motions and transfer reference-like styles across different terrains and velocities, while remaining robust to variations in reward design. We validate the effectiveness of our method through extensive experiments in both simulation and on a Unitree Go2 robot. By leveraging demonstrations to guide exploration during RL training, without imposing explicit bias toward them, APEX enables legged robots to learn with greater stability, efficiency, and generalization. We believe this approach paves the way for guidance-driven RL to boost natural skill acquisition in a wide array of robotic tasks, from locomotion to manipulation. Website and code: https://marmotlab.github.io/APEX/.

Figures (8)

• Figure 1: Illustration of APEX’s decaying action priors: like the braces that stabilize early motion before breaking away, the priors guide exploration at the start of training and fade to zero, enabling a pure RL policy that runs independently at deployment. Images inspired by Forrest Gump (1994).
• Figure 2: Overview of APEX framework. Only dashed lines are required during deployment; 1) Demonstration data can be collected from either a privileged teacher policy, motion capture data, or other sources like animation; 2) Action Priors (sec \ref{['subsec:decap']}) are calculated from demonstration's kinematic joint data and added to the policy actions to bias exploration; 3) Multi-critic (sec \ref{['subsec:multi-critic']}) PPO is used to train the final motion with both style and task+regularization rewards; 4) the trained policy is finally transferred to the hardware zero-shot.
• Figure 3: Visual comparison of motion tracking across training iterations for DeepMimic (DM) and APEX. The APEX policies (blue) demonstrate accurate motion tracking from the early stages, highlighting superior sample efficiency even relative to DM-Full (pink). By $2000$ iterations, both APEX-Full and APEX have effectively converged, whereas DM-Full has converged but failed to learn the task well, as seen in its final poses. All sequences show evaluation policies with action priors set to zero, ensuring fair comparison based purely on learned policies.
• Figure 4: General trend of rewards averaged across seeds for the trot motion. APEX-Full achieves faster convergence than DM-Full, while APEX outperforms DM-NIA without requiring imitation references during deployment. The same trend is observed across other motions.
• Figure 5: APEX' Multi-critic variants sustain high tracking accuracy across all metrics and reward magnitudes, whereas single-critic variants degrade under strong velocity rewards. This highlights how multi-critic learning complements action priors in improving stability and reward robustness.