Task-Centric Policy Optimization from Misaligned Motion Priors

TL;DR

This work addresses the problem of integrating motion priors from human demonstrations into humanoid control when those priors are misaligned with the robot’s task. It introduces Task-Centric Motion Priors (TCMP), a gradient-projection framework that treats imitation as a conditional regularizer rather than a co-equal objective, guaranteeing task-feasible updates that preserve progress on $J_{\text{task}}(\theta)$ while selectively utilizing imitation signals from $J_{\text{style}}(\theta)$. The method provides a closed-form update direction $g^* = \alpha g_{\text{task}}+(1-\alpha) g_{\text{style}}$ with an data-driven $\alpha$ that adapts to gradient geometry, plus a full algorithm built on PPO with two value functions and a discriminator. Theoretical results prove descent guarantees, a notion of task-priority stationarity, and convergence, while experiments on eight Unitree G1 humanoid tasks demonstrate robust task performance and coherent motion across misaligned demonstrations, outperforming standard AMP and task-only baselines without manual tuning. Overall, TCMP offers a principled, interpretable approach to leveraging motion priors in robotic control under imperfect demonstrations, with strong practical implications for robust imitation-learning in real-world humanoid systems.

Abstract

Humanoid control often leverages motion priors from human demonstrations to encourage natural behaviors. However, such demonstrations are frequently suboptimal or misaligned with robotic tasks due to embodiment differences, retargeting errors, and task-irrelevant variations, causing naïve imitation to degrade task performance. Conversely, task-only reinforcement learning admits many task-optimal solutions, often resulting in unnatural or unstable motions. This exposes a fundamental limitation of linear reward mixing in adversarial imitation learning. We propose \emph{Task-Centric Motion Priors} (TCMP), a task-priority adversarial imitation framework that treats imitation as a conditional regularizer rather than a co-equal objective. TCMP maximizes task improvement while incorporating imitation signals only when they are compatible with task progress, yielding an adaptive, geometry-aware update that preserves task-feasible descent and suppresses harmful imitation under misalignment. We provide theoretical analysis of gradient conflict and task-priority stationary points, and validate our claims through humanoid control experiments demonstrating robust task performance with consistent motion style under noisy demonstrations.

Figures (18)

• Figure 1: Task-only optimization yields unnatural behaviors, while motion priors may be misaligned. We resolve this by selecting the best style among task-optimal solutions.
• Figure 2: Framework of TCMP. Instead of linearly combining task and imitation rewards, TCMP treats adversarial imitation as a conditional regularizer guided by optimization geometry. At each update, the policy is optimized primarily for task performance, while imitation gradients are adaptively incorporated only when they do not interfere with task improvement.
• Figure 3: Qualitative comparison of optimization behaviors under demonstration misalignment. Left: Failure modes of AMP-style reward mixing under misaligned demonstrations, including imitation-dominated collapse (e.g., stationary dog-move) and task-only degeneration with physically implausible motions. Right: TCMP results on the same tasks, exhibiting task-centric, physically consistent behaviors in simulation and successful sim-to-real transfer on the Unitree G1 robot.
• Figure 4: Evolution of the adaptive weighting $\alpha$ under different levels of demonstration alignment. When demonstrations are well aligned with the task, $\alpha$ remains stable and close to zero. As alignment degrades, $\alpha$ progressively increases, suppressing imitation updates that conflict with task-centric optimization.
• Figure 5: Ablation study on the role of the adaptive weighting parameter $\alpha$. We compare TCMP with adaptive $\alpha$ against fixed-$\alpha$ variants and AMP with matched reward-level mixing coefficients. Adaptive $\alpha$ enables TCMP to suppress misleading imitation gradients and recover task-centric optimization, while fixed-$\alpha$ methods collapse to AMP-like behavior under demonstration misalignment.

Theorems & Definitions (39)

• Definition 3.1: Gradient Conflict
• Lemma 3.2: Gradient Decomposition under Linear Scalarization
• proof
• Lemma 3.3: First-Order Objective Variation
• proof
• Definition 3.4: Task-optimal policy set
• Proposition 3.5: Imitation distinguishes task-equivalent policies
• proof
• Theorem 3.7: Failure of linear scalarization under misalignment
• proof