Real-Time Synchronized Interaction Framework for Emotion-Aware Humanoid Robots

TL;DR

This work tackles the challenge of real-time, emotionally synchronized co-speech gestures for humanoid robots in social settings. It introduces ReSIn-HR, a three-tier framework that combines a dual-channel emotion engine, duration-aware speech-gesture synchronization, and biomechanical constraint verification, with LLM-guided motion planning and prompt-based gesture synthesis. Empirical results on the NAO robot show a 21% improvement in emotional alignment and strong gains in synchronization and motion naturalness, despite hardware constraints that limit high-frequency motion. The approach enables more natural and context-aware human-robot interaction, with potential impact on healthcare, education, and service platforms, and includes plans to release code and models for community use.

Abstract

As humanoid robots increasingly introduced into social scene, achieving emotionally synchronized multimodal interaction remains a significant challenges. To facilitate the further adoption and integration of humanoid robots into service roles, we present a real-time framework for NAO robots that synchronizes speech prosody with full-body gestures through three key innovations: (1) A dual-channel emotion engine where large language model (LLM) simultaneously generates context-aware text responses and biomechanically feasible motion descriptors, constrained by a structured joint movement library; (2) Duration-aware dynamic time warping for precise temporal alignment of speech output and kinematic motion keyframes; (3) Closed-loop feasibility verification ensuring gestures adhere to NAO's physical joint limits through real-time adaptation. Evaluations show 21% higher emotional alignment compared to rule-based systems, achieved by coordinating vocal pitch (arousal-driven) with upper-limb kinematics while maintaining lower-body stability. By enabling seamless sensorimotor coordination, this framework advances the deployment of context-aware social robots in dynamic applications such as personalized healthcare, interactive education, and responsive customer service platforms.

Figures (6)

• Figure 1: Traditional gesture systems play static actions disconnected from speech timing. ReSIn-HR generates dynamic, emotionally aligned gestures that synchronize with both content and prosody.
• Figure 2: System architecture of ReSIn-HR for multimodal interaction.
• Figure 3: Overview of NAO’s joint structure.
• Figure 4: Mean angular jerk (rad/s$^3$) across emotions. Lower jerk values indicate smoother, more natural gestures.
• Figure 5: Joint motion trajectory analysis across models.