Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FABG : End-to-end Imitation Learning for Embodied Affective Human-Robot Interaction

Yanghai Zhang, Changyi Liu, Keting Fu, Wenbin Zhou, Qingdu Li, Jianwei Zhang

TL;DR

FABG addresses the challenge of end-to-end imitation learning for expressive human-robot interaction by integrating an immersive VR-based demonstration pipeline, depth-enhanced multimodal perception, and a Prediction-Driven Latency Compensation (PDLC) mechanism. By collecting high-quality demonstrations with synchronized operator facial cues and first-person RGB-D views, and fusing semantic and geometric features into a robust 3D perception representation, the approach enables real-time, fluid facial behaviors on a 25-DoF humanoid head. The PDLC strategy eliminates action stuttering and compensates multi-source delays by executing the $n+1$-th action from a fixed-length sequence of length $k$, with per-timestep policy queries. Experimental results across four interaction tasks show significant improvements in response speed and motion fluency for RGB-D inputs with PDLC, validating the practicality of FABG for social robotics and suggesting avenues for future integration with language models for richer human-robot communication.

Abstract

This paper proposes FABG (Facial Affective Behavior Generation), an end-to-end imitation learning system for human-robot interaction, designed to generate natural and fluid facial affective behaviors. In interaction, effectively obtaining high-quality demonstrations remains a challenge. In this work, we develop an immersive virtual reality (VR) demonstration system that allows operators to perceive stereoscopic environments. This system ensures "the operator's visual perception matches the robot's sensory input" and "the operator's actions directly determine the robot's behaviors" - as if the operator replaces the robot in human interaction engagements. We propose a prediction-driven latency compensation strategy to reduce robotic reaction delays and enhance interaction fluency. FABG naturally acquires human interactive behaviors and subconscious motions driven by intuition, eliminating manual behavior scripting. We deploy FABG on a real-world 25-degree-of-freedom (DoF) humanoid robot, validating its effectiveness through four fundamental interaction tasks: expression response, dynamic gaze, foveated attention, and gesture recognition, supported by data collection and policy training. Project website: https://cybergenies.github.io

FABG : End-to-end Imitation Learning for Embodied Affective Human-Robot Interaction

TL;DR

FABG addresses the challenge of end-to-end imitation learning for expressive human-robot interaction by integrating an immersive VR-based demonstration pipeline, depth-enhanced multimodal perception, and a Prediction-Driven Latency Compensation (PDLC) mechanism. By collecting high-quality demonstrations with synchronized operator facial cues and first-person RGB-D views, and fusing semantic and geometric features into a robust 3D perception representation, the approach enables real-time, fluid facial behaviors on a 25-DoF humanoid head. The PDLC strategy eliminates action stuttering and compensates multi-source delays by executing the -th action from a fixed-length sequence of length , with per-timestep policy queries. Experimental results across four interaction tasks show significant improvements in response speed and motion fluency for RGB-D inputs with PDLC, validating the practicality of FABG for social robotics and suggesting avenues for future integration with language models for richer human-robot communication.

Abstract

This paper proposes FABG (Facial Affective Behavior Generation), an end-to-end imitation learning system for human-robot interaction, designed to generate natural and fluid facial affective behaviors. In interaction, effectively obtaining high-quality demonstrations remains a challenge. In this work, we develop an immersive virtual reality (VR) demonstration system that allows operators to perceive stereoscopic environments. This system ensures "the operator's visual perception matches the robot's sensory input" and "the operator's actions directly determine the robot's behaviors" - as if the operator replaces the robot in human interaction engagements. We propose a prediction-driven latency compensation strategy to reduce robotic reaction delays and enhance interaction fluency. FABG naturally acquires human interactive behaviors and subconscious motions driven by intuition, eliminating manual behavior scripting. We deploy FABG on a real-world 25-degree-of-freedom (DoF) humanoid robot, validating its effectiveness through four fundamental interaction tasks: expression response, dynamic gaze, foveated attention, and gesture recognition, supported by data collection and policy training. Project website: https://cybergenies.github.io

Paper Structure

This paper contains 14 sections, 1 equation, 5 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Demonstration Platform
  4. Imitation Learning for Robotic Manipulation
  5. METHOD
  6. Immersive VR Demonstration System
  7. Depth-Augmented Observations
  8. Prediction-Driven Latency Compensation
  9. EXPERIMENT
  10. Hardware Platform
  11. Experimental Setup
  12. Experimental Results
  13. Ablation experiment
  14. CONCLUSIONS

Figures (5)

  • Figure 1: Human-to-robot behavior transfer. Top: VR captures facial landmarks/head pose (operator) and stereo RGB-D (ego-view). Bottom: We use the action chunking transformer to train an imitation policy for each task.
  • Figure 2: Alignment between PICO passthrough view and ZED camera feed for establishing observation space positioning in stereoscopic environments.
  • Figure 3: Our approach employs Action Chunking with PDLC, replacing conventional interleaved observation-execution cycles.
  • Figure 4: Human-Robot Interaction Scenarios in FABG
  • Figure 5: Comparative output curves of the JawOpen parameter generated by three strategies (PDLC, w/ TE, w/o TE) versus human demonstrations under identical model conditions.