Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion

Khang Nguyen, Khai Nguyen, An T. Le, Jan Peters, Manfred Huber, Ngo Anh Vien, Minh Nhat Vu

TL;DR

TD-GRPC tackles instability and policy mismatch in high-dimensional humanoid locomotion by integrating Group Relative Policy Constraint with explicit latent-space trust-region constraints into a Temporal-Difference Model Predictive Control framework. The method leverages latent dynamics, short-horizon planning, and GRPO to guide policy improvement via relative action advantages while limiting distributional drift. Empirical results on the 26-DoF Unitree H1-2 across ten locomotion tasks show faster convergence and improved stability versus SAC, TD-MPC2, and TD-M(PC)^2, though some tasks like crawling and stair-climbing remain challenging. The work demonstrates that constraint-aware TD-MBRL can achieve robust, sample-efficient locomotion, motivating broader use of GRPO-like approaches in high-dimensional control.

Abstract

Robot learning in high-dimensional control settings, such as humanoid locomotion, presents persistent challenges for reinforcement learning (RL) algorithms due to unstable dynamics, complex contact interactions, and sensitivity to distributional shifts during training. Model-based methods, \textit{e.g.}, Temporal-Difference Model Predictive Control (TD-MPC), have demonstrated promising results by combining short-horizon planning with value-based learning, enabling efficient solutions for basic locomotion tasks. However, these approaches remain ineffective in addressing policy mismatch and instability introduced by off-policy updates. Thus, in this work, we introduce Temporal-Difference Group Relative Policy Constraint (TD-GRPC), an extension of the TD-MPC framework that unifies Group Relative Policy Optimization (GRPO) with explicit Policy Constraints (PC). TD-GRPC applies a trust-region constraint in the latent policy space to maintain consistency between the planning priors and learned rollouts, while leveraging group-relative ranking to assess and preserve the physical feasibility of candidate trajectories. Unlike prior methods, TD-GRPC achieves robust motions without modifying the underlying planner, enabling flexible planning and policy learning. We validate our method across a locomotion task suite ranging from basic walking to highly dynamic movements on the 26-DoF Unitree H1-2 humanoid robot. Through simulation results, TD-GRPC demonstrates its improvements in stability and policy robustness with sampling efficiency while training for complex humanoid control tasks.

TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion

TL;DR

TD-GRPC tackles instability and policy mismatch in high-dimensional humanoid locomotion by integrating Group Relative Policy Constraint with explicit latent-space trust-region constraints into a Temporal-Difference Model Predictive Control framework. The method leverages latent dynamics, short-horizon planning, and GRPO to guide policy improvement via relative action advantages while limiting distributional drift. Empirical results on the 26-DoF Unitree H1-2 across ten locomotion tasks show faster convergence and improved stability versus SAC, TD-MPC2, and TD-M(PC)^2, though some tasks like crawling and stair-climbing remain challenging. The work demonstrates that constraint-aware TD-MBRL can achieve robust, sample-efficient locomotion, motivating broader use of GRPO-like approaches in high-dimensional control.

Abstract

Robot learning in high-dimensional control settings, such as humanoid locomotion, presents persistent challenges for reinforcement learning (RL) algorithms due to unstable dynamics, complex contact interactions, and sensitivity to distributional shifts during training. Model-based methods, \textit{e.g.}, Temporal-Difference Model Predictive Control (TD-MPC), have demonstrated promising results by combining short-horizon planning with value-based learning, enabling efficient solutions for basic locomotion tasks. However, these approaches remain ineffective in addressing policy mismatch and instability introduced by off-policy updates. Thus, in this work, we introduce Temporal-Difference Group Relative Policy Constraint (TD-GRPC), an extension of the TD-MPC framework that unifies Group Relative Policy Optimization (GRPO) with explicit Policy Constraints (PC). TD-GRPC applies a trust-region constraint in the latent policy space to maintain consistency between the planning priors and learned rollouts, while leveraging group-relative ranking to assess and preserve the physical feasibility of candidate trajectories. Unlike prior methods, TD-GRPC achieves robust motions without modifying the underlying planner, enabling flexible planning and policy learning. We validate our method across a locomotion task suite ranging from basic walking to highly dynamic movements on the 26-DoF Unitree H1-2 humanoid robot. Through simulation results, TD-GRPC demonstrates its improvements in stability and policy robustness with sampling efficiency while training for complex humanoid control tasks.
Paper Structure (28 sections, 1 theorem, 15 equations, 4 figures, 2 tables, 1 algorithm)

This paper contains 28 sections, 1 theorem, 15 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Difference Learning with Group Relative Policy Constraint
  4. Problem Formulation
  5. TD Learning with Group Relative Policy Constraint
  6. Policy Constraint as Residual Learning
  7. Group Relative Policy Constraint
  8. Experimental Results & Analysis
  9. Quantitative Comparisons
  10. Standing
  11. Walking
  12. Running
  13. Sitting
  14. Crawling
  15. Balancing
  16. ...and 13 more sections

Key Result

Lemma III.1

Suppose we have a value function $V^{\pi}(\textbf{s}) = \mathbb{E}_\pi \left[\sum_{t=0}^{\infty} \gamma^t r(\textbf{s}_t, \textbf{a}_t) \mid \textbf{s}_0 = \textbf{s} \right]$ such that $\max_s|V^*(\textbf{s}) - \hat{V}(\textbf{s})| \leq \varepsilon_v$. The performance of the $1$-step greedy policy

Figures (4)

  • Figure 1: Locomotion Tasks Performed by the Unitree H1-2 Humanoid with TD-GRPC: standing, sitting on a chair, walking, running, crawling under a tunnel, climbing stairs, balancing on a ball-board toy, and navigating through standing poles while avoiding collision.
  • Figure 2: Overview of TD-GRPC for Humanoid Locomotion: Starting from an initial state $\textbf{s}_0$ encoded into latent state $\textbf{z}_0$ with an encoder $h_{\theta}$, a latent dynamics model $d_{\theta}$ takes an action $\textbf{a}_t$ and the latent state $\textbf{z}_t$ to predict the next latent state $\textbf{z}_{t+1}$ across $H$ steps of MPPI planning horizon. In each step, the reward, $Q$-value, and action are estimated via the MLPs $R_{\theta}$, $Q_{\theta}$, and $\pi_{\theta}$, enabling latent-space planning with TD targets. At each state, sampled groups of actions are rolled out to evaluate $Q$-values, which are used to compute softmax-based advantage scores $A^{g}$ of the $g^{th}$ group. These scores are then used in the GRPC objective to guide the policy toward high-value actions while minimizing variance. To prevent excessive policy shifts, a trust-region constraint is imposed via a KL divergence penalty between the current and a prior MPPI-derived policies, enforcing residual learning with bounded policy divergence.
  • Figure 3: Episode Returns of TD-GRPC and Baselines on H1–2 in Humanoid Locomotion Tasks: TD-GRPC achieves rapid convergence over others in standing, walking, running, sitting, navigating through poles, hurdling, and sliding tasks, while it performs worse in crawling tasks. In general, TD-GRPC shows slightly better data-efficiency than TD-M(PC)$^2$ and significantly better sampling-efficiency than SAC and TD-MPC2, alongside the fact that TD-GRPC outperforms TD-MPC2 and SAC on many tasks quantitatively. Nevertheless, all benchmarked algorithms fail to accomplish more challenging tasks, such as stair-climbing and balancing on a ball-board platform.
  • Figure 4: Behavioral Analysis of H1-2 in Humanoid Locomotion Tasks: (a) Walking and Running Backwards: walking direction comparisons between TD-MPC2, TD-M(PC)$^2$, and our approach. TD-GRPC directs H1-2 to move forward, but TD-MPC2 and TD-M(PC)$^2$ make it walk/run backward. (b) Sitting Stability: sitting pose comparisons between TD-MPC2, TD-M(PC)$^2$, and TD-GRPC. TD-GRPC achieves optimal sitting leg pose. All frames are taken from the last state of the evaluation episode. (c) Crawling Pose and Height: The crawling pose and height produced by TD-MPC2 are better than those generated by TD-M(PC)$^2$ and TD-GRPC. (d) Balancing is Hard for Heavy Body: Due to its heavy body weight, all three methods suffer difficulties in keeping H1-2 balance itself on the platform. However, TD-GRPC can balance the robot for a short period; meanwhile, TD-MPC2 and TD-M(PC)$^2$ make the robot's legs flick the board away and fail immediately. (e) Navigating through Standing Poles: TD-M(PC)$^2$ and TD-GRPC induce a standing behavior without navigating, while SAC and TD-MPC2 produce valid motions but collide with poles and fail. (f) Arm-Balancing Helps Locomotion: Freezing the upper body of H1-2 makes walking and running unstable. This experimental finding is evaluated with both TD-M(PC)$^2$ and TD-GRPC.

Theorems & Definitions (1)

  • Lemma III.1: Singh and Yee singh1994upper