Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation

Yara Mahmoud, Yasheerah Yaqoot, Miguel Altamirano Cabrera, Dzmitry Tsetserukou

TL;DR

HumanoidVLM addresses the gap between semantic scene understanding and low-level impedance control in humanoid manipulation by linking an egocentric image to task-specific Cartesian impedance parameters $K=[K_x,K_y,K_z]$, $D=[D_x,D_D_y,D_z]$ and gripper angle $\gamma$ via a vision-language model and a FAISS-based Retrieval-Augmented Generation (RAG) module. Retrieved parameters feed a task-space impedance controller to generate end-effector references that are realized through inverse kinematics on the Unitree G1. The approach achieves a retrieval accuracy of 93% across 14 visual scenarios and demonstrates stable, compliant real-world interaction with $z$-axis tracking errors in the range of $1$–$3.5$ cm, illustrating a feasible, interpretable path toward adaptive humanoid manipulation. By grounding semantic perception in experimentally validated control parameters, HumanoidVLM provides a principled bridge from perception to safe, task-aware compliant behavior in contact-rich tasks.

Abstract

Humanoid robots must adapt their contact behavior to diverse objects and tasks, yet most controllers rely on fixed, hand-tuned impedance gains and gripper settings. This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate Cartesian impedance parameters and gripper configurations directly from an egocentric RGB image. The system couples a vision-language model for semantic task inference with a FAISS-based Retrieval-Augmented Generation (RAG) module that retrieves experimentally validated stiffness-damping pairs and object-specific grasp angles from two custom databases, and executes them through a task-space impedance controller for compliant manipulation. We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%. Real-world experiments show stable interaction dynamics, with z-axis tracking errors typically within 1-3.5 cm and virtual forces consistent with task-dependent impedance settings. These results demonstrate the feasibility of linking semantic perception with retrieval-based control as an interpretable path toward adaptive humanoid manipulation.

HumanoidVLM: Vision-Language-Guided Impedance Control for Contact-Rich Humanoid Manipulation

TL;DR

HumanoidVLM addresses the gap between semantic scene understanding and low-level impedance control in humanoid manipulation by linking an egocentric image to task-specific Cartesian impedance parameters , and gripper angle via a vision-language model and a FAISS-based Retrieval-Augmented Generation (RAG) module. Retrieved parameters feed a task-space impedance controller to generate end-effector references that are realized through inverse kinematics on the Unitree G1. The approach achieves a retrieval accuracy of 93% across 14 visual scenarios and demonstrates stable, compliant real-world interaction with -axis tracking errors in the range of – cm, illustrating a feasible, interpretable path toward adaptive humanoid manipulation. By grounding semantic perception in experimentally validated control parameters, HumanoidVLM provides a principled bridge from perception to safe, task-aware compliant behavior in contact-rich tasks.

Abstract

Humanoid robots must adapt their contact behavior to diverse objects and tasks, yet most controllers rely on fixed, hand-tuned impedance gains and gripper settings. This paper introduces HumanoidVLM, a vision-language driven retrieval framework that enables the Unitree G1 humanoid to select task-appropriate Cartesian impedance parameters and gripper configurations directly from an egocentric RGB image. The system couples a vision-language model for semantic task inference with a FAISS-based Retrieval-Augmented Generation (RAG) module that retrieves experimentally validated stiffness-damping pairs and object-specific grasp angles from two custom databases, and executes them through a task-space impedance controller for compliant manipulation. We evaluate HumanoidVLM on 14 visual scenarios and achieve a retrieval accuracy of 93%. Real-world experiments show stable interaction dynamics, with z-axis tracking errors typically within 1-3.5 cm and virtual forces consistent with task-dependent impedance settings. These results demonstrate the feasibility of linking semantic perception with retrieval-based control as an interpretable path toward adaptive humanoid manipulation.
Paper Structure (17 sections, 4 equations, 3 figures, 1 table)

This paper contains 17 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Works
  3. System Architecture
  4. VLM--RAG System
  5. Custom Database for Environmental Scenarios
  6. Robot Control
  7. End-effector Impedance Control
  8. Gripper Configuration
  9. System Evaluation
  10. Experimental Setup
  11. VLM--RAG Retrieval Evaluation
  12. Retrieval Process
  13. Impedance Control Evaluation
  14. Results
  15. VLM--RAG System
  16. ...and 2 more sections

Figures (3)

  • Figure 1: Images from successful trials of the representative manipulation scenarios.
  • Figure 2: Retrieval accuracy of the VLM--RAG system across 14 scenarios.
  • Figure 3: Translational errors in the surface-following task, showing desired and measured end-effector positions using forward kinematics.