Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Wenqi Cai, Kyriakos G. Vamvoudakis, Sébastien Gros, Anthony Tzes

Abstract

In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.

Cost-Matching Model Predictive Control for Efficient Reinforcement Learning in Humanoid Locomotion

Abstract

In this paper, we propose a cost-matching approach for optimal humanoid locomotion within a Model Predictive Control (MPC)-based Reinforcement Learning (RL) framework. A parameterized MPC formulation with centroidal dynamics is trained to approximate the action-value function obtained from high-fidelity closed-loop data. Specifically, the MPC cost-to-go is evaluated along recorded state-action trajectories, and the parameters are updated to minimize the discrepancy between MPC-predicted values and measured returns. This formulation enables efficient gradient-based learning while avoiding the computational burden of repeatedly solving the MPC problem during training. The proposed method is validated in simulation using a commercial humanoid platform. Results demonstrate improved locomotion performance and robustness to model mismatch and external disturbances compared with manually tuned baselines.

Paper Structure

This paper contains 24 sections, 1 theorem, 21 equations, 5 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Centroidal Dynamics
  4. Constraints
  5. Inequality Constraints
  6. Equality Constraints
  7. Optimal Control Objective
  8. Cost-Matching MPC-Based Reinforcement Learning
  9. Parameterized MPC as a Q Function
  10. Parameterized MPC scheme
  11. MPC-induced action-value function
  12. Action-Value Evaluation via Data Rollouts
  13. Cost-Matching Objective and Gradient
  14. Convergence of Cost Matching
  15. Constraints and Feasibility
  16. ...and 9 more sections

Key Result

Theorem 1

Suppose that Assumption 1 holds and the loss $\mathcal{L}$ in eq:cm_loss has Lipschitz-continuous gradient with constant $L_{\mathcal{L}}=2(G_Q^2+B_QL_Q)$. Let the parameter sequence $\{\boldsymbol{\mathrm{\theta}}_j\}$ be generated by $\boldsymbol{\mathrm{\theta}}_{j+1}=\boldsymbol{\mathrm{\theta}} where $\mathcal{L}_{\inf}:=\inf_{\boldsymbol{\mathrm{\theta}}}\mathcal{L}(\boldsymbol{\mathrm{\thet

Figures (5)

  • Figure 2: Cost-Matching MPC-RL framework for humanoids.
  • Figure 3: Closed-loop validation of the value mismatch $\mathrm{MSE}\!\left(Q^{\mathrm{MPC}}_{\boldsymbol{\mathrm{\theta}}}-Q^{\mathrm{meas}}\right)$ across training rounds (shaded region: $10-90$% across seeds). At each training round, $50$ trajectories are collected ($600$ steps per trajectory), and the cost-matching objective is optimized using $2000$ gradient updates with discount factor $\gamma=0.985$. The inset summarizes the normalized block-wise evolution of $\boldsymbol{\mathrm{\theta}}$, illustrating how learning reshapes the MPC objective.
  • Figure 4: Value-matching diagnostics on a fixed evaluation set: density plot of $Q^{\mathrm{MPC}}_{\boldsymbol{\mathrm{\theta}}}$ versus $Q^{\mathrm{meas}}$.
  • Figure 5: Simulation snapshots of the humanoid during locomotion under a time-scheduled disturbance benchmark consisting of lateral force pulses $f_{\mathrm{ext},y}$ and a yaw-torque pulse $m_{\mathrm{ext},z}$.
  • Figure 6: Representative closed-loop response under the disturbance benchmark. Left: full-horizon signals. Right: zoom around the push interval. The gray bands indicate push windows. The dashed horizontal line indicates the settle threshold computed from pre-disturbance statistics, and the "settled" marker indicates the first time the error remains below this threshold.

Theorems & Definitions (2)

  • Theorem 1: Convergence of cost matching
  • proof