HMC: Learning Heterogeneous Meta-Control for Contact-Rich Loco-Manipulation

TL;DR

The paper addresses the challenge of contact-rich loco-manipulation where purely position-based policies underperform. It introduces Heterogeneous Meta-Control (HMC), comprising an HMC-Controller that blends torque commands from multiple control modalities and an HMC-Policy that learns a heterogeneous set of experts via a two-stage pretrain-finetune strategy with soft routing. The approach uses a shared Transformer trunk and modality-specific heads, enabling smooth transitions between modes and robust whole-body coordination, and is trained to leverage large-scale position data alongside fine-grained force demonstrations. Real-world experiments on a Unitree G1 show substantial improvements in stability, compliance, and generalization across tasks like wiping, lifting, and drawer opening, highlighting the practical impact of adaptive, torque-space control in loco-manipulation.

Abstract

Learning from real-world robot demonstrations holds promise for interacting with complex real-world environments. However, the complexity and variability of interaction dynamics often cause purely positional controllers to struggle with contacts or varying payloads. To address this, we propose a Heterogeneous Meta-Control (HMC) framework for Loco-Manipulation that adaptively stitches multiple control modalities: position, impedance, and hybrid force-position. We first introduce an interface, HMC-Controller, for blending actions from different control profiles continuously in the torque space. HMC-Controller facilitates both teleoperation and policy deployment. Then, to learn a robust force-aware policy, we propose HMC-Policy to unify different controllers into a heterogeneous architecture. We adopt a mixture-of-experts style routing to learn from large-scale position-only data and fine-grained force-aware demonstrations. Experiments on a real humanoid robot show over 50% relative improvement vs. baselines on challenging tasks such as compliant table wiping and drawer opening, demonstrating the efficacy of HMC.

Figures (4)

• Figure 1: System overview. HMC-Controller accepts inputs from either a VR-based teleoperation system or HMC-Policy inference. In the model inference path, multiple expert heads output corresponding control strategies. In the teleoperation path, shared hand poses are distributed to the expert controllers with different controller-specific parameters. All expert controllers output joint-space torque commands, which are blended via soft routing using predicted soft weights. Finally, the fused torque commands and lower-body joint targets are executed by the robot in real time. ("J.S": Joint Space. "C.S": Cartesian Space.)
• Figure 2: Overview of Two-stage HMC.(a) Pretraining: We harness abundant positional demonstrations to train the shared transformer trunk and position expert head, thereby embedding a strong positional prior that boosts generalization. (b) Fine-tuning Stage: All parameters are unfrozen and fine-tuned on a smaller, fine-grained multi-expert dataset. A soft routing network learns to blend outputs from multiple experts, producing smooth and adaptive control policies. ("J.S": Joint Space. "C.S": Cartesian Space.)
• Figure 3: Tasks visualization.
• Figure 4: Interpretability of routing logits in visualization of an Open Drawer Episode. The upper figure is the predicted routing logits, while the lower figure shows the predicted right-hand stiffness ($N/m$). Five images from the head camera are demonstrated.