Residual MPC: Blending Reinforcement Learning with GPU-Parallelized Model Predictive Control

TL;DR

This work addresses the robustness gap between model-based MPC and model-free RL in legged locomotion by embedding a GPU-parallelized kinodynamic whole-body MPC as a fixed control prior within an RL loop. A residual policy is trained to apply targeted torque-level corrections to the MPC outputs at 100 Hz, enabling in-the-loop learning across thousands of parallel agents. The approach yields higher sample efficiency, larger achievable velocity commands, improved self-collision handling, and zero-shot adaptability to unseen gaits and uneven terrain, with hardware validation on MIT Humanoid. Despite increased wall-clock training time due to MPC evaluation, the resulting policy demonstrates safer, more hardware-friendly locomotion and points to future directions in learning MPC parameters and uncertainty-aware blending.

Abstract

Model Predictive Control (MPC) provides interpretable, tunable locomotion controllers grounded in physical models, but its robustness depends on frequent replanning and is limited by model mismatch and real-time computational constraints. Reinforcement Learning (RL), by contrast, can produce highly robust behaviors through stochastic training but often lacks interpretability, suffers from out-of-distribution failures, and requires intensive reward engineering. This work presents a GPU-parallelized residual architecture that tightly integrates MPC and RL by blending their outputs at the torque-control level. We develop a kinodynamic whole-body MPC formulation evaluated across thousands of agents in parallel at 100 Hz for RL training. The residual policy learns to make targeted corrections to the MPC outputs, combining the interpretability and constraint handling of model-based control with the adaptability of RL. The model-based control prior acts as a strong bias, initializing and guiding the policy towards desirable behavior with a simple set of rewards. Compared to standalone MPC or end-to-end RL, our approach achieves higher sample efficiency, converges to greater asymptotic rewards, expands the range of trackable velocity commands, and enables zero-shot adaptation to unseen gaits and uneven terrain.

Figures (17)

• Figure 1: Simulated locomotion for MPC (left, grey) and the residual policy (right, orange) at velocity commands of 2.25 m/s and -1.5 rad/s. The MPC controller fails and terminates from self-collision, while the residual policy is able to stabilize the robot. We use Viser for 3D visualization in this work yi2025viser.
• Figure 2: Diagram of the proposed residual policy architecture during training and deployment. Both the multilayer perceptron (MLP) policy and MPC are evaluated at the control frequency of 100 Hz, and are blended downstream with a scaling factor of $\lambda$ on the policy output. The MPC solution is converted into desired torques, joint positions, and joint velocities with the inverse dynamics (ID) of the system. We efficiently evaluate the MPC controller across thousands of instances on the GPU for concurrent training with the residual network.
• Figure 3: Visualized predictions for the MPC formulation (distances are adjusted for visual clarity). A Bezier curve is defined by the peak swing height, takeoff velocity, and touchdown velocity. This curve is parametrized by the contact phase to output desired foot heights.
• Figure 4: Plot of survival rate vs. number of QP iterations for MPC environments subject to randomized initial velocities in the range $\left\lVert\hat{\mathop{\mathrm{\mathbf{v}}}\nolimits}_{x, y}\right\rVert \leq 0.5$ m/s and $\left\lVert\hat{\boldsymbol{\omega}}\right\rVert \leq 0.5$ rad/s. Beyond a certain number of iterations, the marginal benefit is negligble while increasing computation time.
• Figure 5: Computation time required for evaluating the full MPC controller on the CPU and GPU across 1000 environments in IsaacGym. We optimistically estimate the required time for the CPU assuming 10 perfectly parallel cores, and do not include the time necessary for CPU-GPU data transfer. More detailed benchmarking results for CusADi on the CPU and GPU are shown in jeon2024cusadi.