RHINO: Learning Real-Time Humanoid-Human-Object Interaction from Human Demonstrations

TL;DR

RHINO tackles real-time humanoid–human–object interaction by learning a two-level policy: a high-frequency reactive planner infers $I_t \in \mathcal{I}$ at 30 Hz and a low-level controller executes $K_t \in \mathcal{K}$ with interruptibility. It combines multi-modal observations $\mathcal{O}=\mathcal{E}\odot\mathcal{H}$ and learns from human–object–human demonstrations plus teleoperation data, deploying on a Unitree H1. The framework comprises a reactive planner, diffusion-based interactive motion skills, discrete manipulation skills via ACT, a safety supervisor, and a real-robot platform, demonstrated across dining and office scenarios with 20+ tasks. Results show improved intention recognition, motion realism, manipulation robustness, and safety in real-world conditions, supporting real-world daily-life assistance with real-time responsiveness and safe interruption capabilities.

Abstract

Humanoid robots have shown success in locomotion and manipulation. Despite these basic abilities, humanoids are still required to quickly understand human instructions and react based on human interaction signals to become valuable assistants in human daily life. Unfortunately, most existing works only focus on multi-stage interactions, treating each task separately, and neglecting real-time feedback. In this work, we aim to empower humanoid robots with real-time reaction abilities to achieve various tasks, allowing human to interrupt robots at any time, and making robots respond to humans immediately. To support such abilities, we propose a general humanoid-human-object interaction framework, named RHINO, i.e., Real-time Humanoid-human Interaction and Object manipulation. RHINO provides a unified view of reactive motion, instruction-based manipulation, and safety concerns, over multiple human signal modalities, such as languages, images, and motions. RHINO is a hierarchical learning framework, enabling humanoids to learn reaction skills from human-human-object demonstrations and teleoperation data. In particular, it decouples the interaction process into two levels: 1) a high-level planner inferring human intentions from real-time human behaviors; and 2) a low-level controller achieving reactive motion behaviors and object manipulation skills based on the predicted intentions. We evaluate the proposed framework on a real humanoid robot and demonstrate its effectiveness, flexibility, and safety in various scenarios.

Figures (11)

• Figure 1: RHINO has the capabilities of real-time interaction on diverse tasks. (a) RHINO enables real-time humanoid-human-object interaction, allowing seamless task interruption and dynamic switching during operation. (b) The system demonstrates diverse capabilities, including waving, cheering, stamping, object pickup, handovers, and more.
• Figure 2: Components of the RHINO framework and related works. (a) Illustration of the RHINO framework, including the reactive planner, motion generation skills, object interaction skills and safety supervisor. (b) Classification of related works based on the components of the RHINO framework.
• Figure 3: Network architecture of RHINO modules, including the reactive planner, motion generation, and manipulation skills.
• Figure 4: Information collected in the dataset and the requirement of real robot deployment. The left box shows our Human-Object-Human Interaction Demonstration Data, which contains human motion and hand position information collected from RGB-D images and a simple motion capture system. The box in the middle illustrates our Manipulation Demonstration Data, which is collected through teleoperation. The right box shows the information required in real robot deployment.
• Figure 5: The humanoid robot platform.