Data Efficient Reinforcement Learning for Legged Robots

TL;DR

The paper tackles the data efficiency challenge of legged locomotion by adopting a model-based reinforcement learning framework that learns a long-horizon dynamics model and uses model-predictive control for real-time planning. Key innovations include a multi-step loss to reduce prediction drift, a planning loop implemented with parallelized cross-entropy method, asynchronous replanning to cope with latency, and trajectory generators to constrain exploration for safety. On a Minitaur quadruped, the approach achieves walking with only 4.5 minutes of real-world data and demonstrates zero-shot generalization to unseen tasks like turning and backward walking, outperforming model-free baselines in sample efficiency. These results suggest that combining long-horizon dynamics with latency-aware planning and safety priors yields fast, robust, and generalizable locomotion control suitable for real robots.

Abstract

We present a model-based framework for robot locomotion that achieves walking based on only 4.5 minutes (45,000 control steps) of data collected on a quadruped robot. To accurately model the robot's dynamics over a long horizon, we introduce a loss function that tracks the model's prediction over multiple timesteps. We adapt model predictive control to account for planning latency, which allows the learned model to be used for real time control. Additionally, to ensure safe exploration during model learning, we embed prior knowledge of leg trajectories into the action space. The resulting system achieves fast and robust locomotion. Unlike model-free methods, which optimize for a particular task, our planner can use the same learned dynamics for various tasks, simply by changing the reward function. To the best of our knowledge, our approach is more than an order of magnitude more sample efficient than current model-free methods.

Figures (7)

• Figure 1: Overview of our learning system on the robot. The system alternates between collecting trajectories and learning a dynamics model.
• Figure 2: Timing diagram of our asynchronous controller. (a) The planner outputs $\bm{a}_t, ..., \bm{a}_{t+H}$ given the current state $\bm{s}_t$. However, due to the planning latency $T$, the action $\bm{a}_t$ is actually executed at time $t+T$, on the unplanned state $\bm{s}_{t+T}$, which leads to a suboptimal action. (b) Our system predicts the state $\hat{\bm{s}}_{t+T}$ when the planning completes, and uses it as the input to the planner. As a result, the planned action $\bm{a}_{t+T}$ is executed on the right state $\bm{s}_{t+T}$.
• Figure 3: Illustration of TGs and their interaction with the planner.
• Figure 4: Learning on real robot. (\ref{['fig:real_robot_behavior']}) The robot walks on different terrains. (\ref{['fig:real_robot_speed_profile']}) The robot gradually tracks the desired speed profile. (\ref{['fig:real_robot_gait_pattern_final']}) Swing angles and gait pattern of all four legs. (\ref{['fig:real_robot_perturbation']}) Robot trajectory when the robot walks up and down a slope.
• Figure 5: Generalization of MPC to unseen reward functions using the existing dynamics model. In both cases, the dynamics model is trained only on the task of forward-walking. In \ref{['fig:generalization_backward']}, the new cost function is to track a desired backward speed. In \ref{['fig:generalization_turn']}, the new cost function is to keep the same forward speed while turning left or right at a rate of 15 degrees per second.