PILOT: A Perceptive Integrated Low-level Controller for Loco-manipulation over Unstructured Scenes

TL;DR

PILOT tackles the challenge of robust loco-manipulation in unstructured environments by unifying perceptive locomotion with large-workspace whole-body control within a single reinforcement learning policy. It introduces a cross-modal context encoder that fuses prediction-based proprioception with an attention-based perception stream, and a Mixture-of-Experts policy to coordinate diverse motor skills, guided by a LiDAR-based elevation map and a curriculum that progressively expands command complexity. The approach achieves superior command tracking, terrain traversability, and stability in both simulation and real-world tests on the Unitree G1, including stair and platform traversal with payloads and autonomous Lift-Box tasks via hierarchical planning. These results demonstrate PILOT’s potential as a foundational low-level controller for humanoid loco-manipulation in complex, three-dimensional scenes, reducing dependence on teleoperation and motion capture while enabling scalable, robust behaviors. Overall, the work advances perceptive, unified control for humanoids by tightly integrating terrain understanding, predictive state estimation, and coordinated motor control into a single policy framework.

Abstract

Humanoid robots hold great potential for diverse interactions and daily service tasks within human-centered environments, necessitating controllers that seamlessly integrate precise locomotion with dexterous manipulation. However, most existing whole-body controllers lack exteroceptive awareness of the surrounding environment, rendering them insufficient for stable task execution in complex, unstructured scenarios.To address this challenge, we propose PILOT, a unified single-stage reinforcement learning (RL) framework tailored for perceptive loco-manipulation, which synergizes perceptive locomotion and expansive whole-body control within a single policy. To enhance terrain awareness and ensure precise foot placement, we design a cross-modal context encoder that fuses prediction-based proprioceptive features with attention-based perceptive representations. Furthermore, we introduce a Mixture-of-Experts (MoE) policy architecture to coordinate diverse motor skills, facilitating better specialization across distinct motion patterns. Extensive experiments in both simulation and on the physical Unitree G1 humanoid robot validate the efficacy of our framework. PILOT demonstrates superior stability, command tracking precision, and terrain traversability compared to existing baselines. These results highlight its potential to serve as a robust, foundational low-level controller for loco-manipulation in unstructured scenes.

Figures (4)

• Figure 1: Method overview of PILOT. We propose a unified single-stage reinforcement learning framework that seamlessly integrates perceptive locomotion and adaptive whole-body control with an expanded workspace into a single policy. Within this framework, we leverage cross-modal context embeddings—fusing prediction-based proprioceptive features with multi-scale attention-based perceptive representations—to enhance the stability of terrain traversal. These embeddings are then fed into a MoE-based unified actor network to coordinate distinct motion skills. The resulting action outputs are converted into motor torques via a PD controller. For real-world deployment, a LiDAR-based robot-centric elevation map is utilized to capture local terrain geometry.
• Figure 2: Visualization of expert activation across six motion modes. The color intensity represents the magnitude of the activation probability, where deep red denotes higher activation and deep blue indicates lower activation.
• Figure 3: Real-world Experiments.PILOT successfully executes object transport tasks across challenging terrains. The robot is shown traversing (a) a staircase and (b) a high platform while carrying a payload.
• Figure 4: Autonomous policy execution. With PILOT serving as the low-level tracking controller, the high-level policy directs the robot to approach the target box, grasp and lift the payload, and recover to an upright configuration.