Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation

Runpei Dong, Ziyan Li, Xialin He, Saurabh Gupta

TL;DR

This work introduces HERO, a residual-aware end-effector tracking framework for humanoid loco-manipulation that blends IK, motion planning, and learned residual forward models to achieve precise whole-body end-effector control. By decoupling perception and action and leveraging open-vocabulary vision models, HERO enables robust manipulation of novel objects in novel environments with a real-world success rate of 83.8% for reaching and picking up unseen objects. Core contributions include learned residual forward kinematics η and base odometry ξ, a PPO-trained tracking policy π_t, and replanning and goal-adjustment mechanisms, all validated through extensive real-world and simulation experiments showing superior EE accuracy and meaningful workspace expansion via waist bending. The modular design and residual learning approach promote scalable, open-world humanoid manipulation, with strong implications for deploying perception-driven planning and robust control across diverse tasks and environments.

Abstract

Visual loco-manipulation of arbitrary objects in the wild with humanoid robots requires accurate end-effector (EE) control and a generalizable understanding of the scene via visual inputs (e.g., RGB-D images). Existing approaches are based on real-world imitation learning and exhibit limited generalization due to the difficulty in collecting large-scale training datasets. This paper presents a new paradigm, HERO, for object loco-manipulation with humanoid robots that combines the strong generalization and open-vocabulary understanding of large vision models with strong control performance from simulated training. We achieve this by designing an accurate residual-aware EE tracking policy. This EE tracking policy combines classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, c) goal adjustment, and d) replanning. Together, these innovations help us cut down the end-effector tracking error by 3.2x. We use this accurate end-effector tracker to build a modular system for loco-manipulation, where we use open-vocabulary large vision models for strong visual generalization. Our system is able to operate in diverse real-world environments, from offices to coffee shops, where the robot is able to reliably manipulate various everyday objects (e.g., mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests in simulation and the real world demonstrate the effectiveness of our proposed design. We believe the advances in this paper can open up new ways of training humanoid robots to interact with daily objects.

Learning Humanoid End-Effector Control for Open-Vocabulary Visual Loco-Manipulation

TL;DR

This work introduces HERO, a residual-aware end-effector tracking framework for humanoid loco-manipulation that blends IK, motion planning, and learned residual forward models to achieve precise whole-body end-effector control. By decoupling perception and action and leveraging open-vocabulary vision models, HERO enables robust manipulation of novel objects in novel environments with a real-world success rate of 83.8% for reaching and picking up unseen objects. Core contributions include learned residual forward kinematics η and base odometry ξ, a PPO-trained tracking policy π_t, and replanning and goal-adjustment mechanisms, all validated through extensive real-world and simulation experiments showing superior EE accuracy and meaningful workspace expansion via waist bending. The modular design and residual learning approach promote scalable, open-world humanoid manipulation, with strong implications for deploying perception-driven planning and robust control across diverse tasks and environments.

Abstract

Visual loco-manipulation of arbitrary objects in the wild with humanoid robots requires accurate end-effector (EE) control and a generalizable understanding of the scene via visual inputs (e.g., RGB-D images). Existing approaches are based on real-world imitation learning and exhibit limited generalization due to the difficulty in collecting large-scale training datasets. This paper presents a new paradigm, HERO, for object loco-manipulation with humanoid robots that combines the strong generalization and open-vocabulary understanding of large vision models with strong control performance from simulated training. We achieve this by designing an accurate residual-aware EE tracking policy. This EE tracking policy combines classical robotics with machine learning. It uses a) inverse kinematics to convert residual end-effector targets into reference trajectories, b) a learned neural forward model for accurate forward kinematics, c) goal adjustment, and d) replanning. Together, these innovations help us cut down the end-effector tracking error by 3.2x. We use this accurate end-effector tracker to build a modular system for loco-manipulation, where we use open-vocabulary large vision models for strong visual generalization. Our system is able to operate in diverse real-world environments, from offices to coffee shops, where the robot is able to reliably manipulate various everyday objects (e.g., mugs, apples, toys) on surfaces ranging from 43cm to 92cm in height. Systematic modular and end-to-end tests in simulation and the real world demonstrate the effectiveness of our proposed design. We believe the advances in this paper can open up new ways of training humanoid robots to interact with daily objects.
Paper Structure (50 sections, 8 equations, 17 figures, 8 tables)

This paper contains 50 sections, 8 equations, 17 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Legged Loco-Manipulation
  4. Loco-manipulation via motion tracking
  5. Visual loco-manipulation
  6. System Identification
  7. HERO: Humanoid End-Effector ContROl
  8. Whole-body End-Effector Tracking Policy, $\pi_\text{t}\xspace$
  9. Residual-aware end-effector tracking
  10. Policy architecture
  11. Training
  12. Learned Residual Neural Forward Models
  13. Residual neural FK
  14. Residual Neural Leg Odometry
  15. Policy Architecture
  16. ...and 35 more sections

Figures (17)

  • Figure 1: HERO is an accurate end-effector control framework. Given an EE goal pose, HERO first uses IK to convert it into an upper-body goal. It then uses motion planning to generate an upper-body reference trajectory that is tracked via a learned tracking policy $\pi_\text{t}\xspace$ (\ref{['sec:method']}). In addition to reference joints, $\pi_\text{t}\xspace$ also takes accurate estimates of the residual EE error (obtained via a learned neural forward model (\ref{['sec:forward']}, \ref{['sec:neural-odometry']}). HERO also employs periodic replanning (\ref{['sec:replan']}) to adapt to drifts and goal adjustment (\ref{['sec:adjustment']}) to mitigate systematic tracking errors. Accurate tracking enables building modular object manipulation systems (\ref{['sec:modular']}, \ref{['fig:system']}).
  • Figure 2: Overall architecture for our proposed modular system for open-vocabulary object grasping. Given a free-form natural language text query indicating which object needs to be picked, we use open-vocabulary large vision models (LVMs like Grounding DINO GroundingDINO24 and SAM SAM325) to segment out the object of interest and predict parallel jaw grasps (using the AnyGrasp model AnyGrasp23). We retarget the predicted grasp to a Dex-3 hand. We use our proposed whole-body end-effector tracker to convey the robot arm to the predicted grasp before picking up the object. By decomposing action planning ( i.e. identifying which object to pick and using what grasp) from action execution ( i.e. actual control of the robot), we inherit the strong visual generalization from pre-trained models as well as strong control capabilities for simulated training of the tracking policy.
  • Figure 3: Learned neural forward kinematics model and odometry model. (a) To correct for inaccurate analytical forward kinematics (FK) that maps joint angles and robot geometry to end-effector poses, we design a residual neural forward kinematics model (\ref{['sec:forward']}), $\eta$, that predicts corrections $\Delta{\bf t}^\text{EE}_t$ & $\Delta{\bf R}^\text{EE}_t$ to the analytical forward kinematics output. (b) As the humanoid's base moves around while reaching the object, the object can go out of view due to large whole-body motions, making closed-loop adjustment from vision infeasible. Thus, it is necessary to accurately estimate base motion. (c) Our residual neural odometry model accurately estimates base odometry from lower body joint states and by assuming that feet remain fixed (\ref{['sec:neural-odometry']}).
  • Figure 4: Novel test environments and novel test objects used for end-to-end testing of our proposed humanoid open-vocabulary object grasping system. (a-b) Standard table ($0.74\text{m}$) and short table ($0.56\text{m}$) setups; note that for short table the robot would first squat down by $15\text{cm}$ the motion planner suggested. (c) 10 daily objects with different shapes, physical properties, appearances, etc (details can be found in \ref{['app:testing_assets']}).
  • Figure 5: Success rate for the end-to-end open-vocabulary grasping task on novel objects in (a) broader and (b) cluttered novel scenes in the real world. (a) We test HERO in 10 daily scenes on 10 new daily objects, such as office lounge and coffee shops. HERO achieves an overall 22/30 (73.3%) success rate, demonstrating strong scene generalization capability. Details of tested locations can be found in \ref{['app:testing_scenes']}. (b) We also test HERO in 5 random cluttered scenes with different layouts. HERO achieves an overall 12/15 (80%) success rate, demonstrating the generalization capability in using language as an accurate proxy for manipulating objects in cluttered scenes.
  • ...and 12 more figures