UGotMe: An Embodied System for Affective Human-Robot Interaction

TL;DR

The paper introduces UGotMe, an embodied system for affective human-robot interaction designed for multiparty conversations. It tackles environmental visual noise and real-time requirements with two denoising strategies and an efficient on-edge data transmission workflow, deploying on Ameca to demonstrate real-time inference. Central to the approach is the Vision-Language to Emotion (VL2E) model, which fuses face-sequence visual cues with conversation context via a crossmodal transformer and SimCSE-based text encoding, achieving state-of-the-art results on the MELD dataset. Real-world experiments show that UGotMe, particularly with VL2E, provides accurate emotional responses and positive user experiences, validating the practicality of denoising and edge-cloud collaboration for affective HRI.

Abstract

Equipping humanoid robots with the capability to understand emotional states of human interactants and express emotions appropriately according to situations is essential for affective human-robot interaction. However, enabling current vision-aware multimodal emotion recognition models for affective human-robot interaction in the real-world raises embodiment challenges: addressing the environmental noise issue and meeting real-time requirements. First, in multiparty conversation scenarios, the noises inherited in the visual observation of the robot, which may come from either 1) distracting objects in the scene or 2) inactive speakers appearing in the field of view of the robot, hinder the models from extracting emotional cues from vision inputs. Secondly, realtime response, a desired feature for an interactive system, is also challenging to achieve. To tackle both challenges, we introduce an affective human-robot interaction system called UGotMe designed specifically for multiparty conversations. Two denoising strategies are proposed and incorporated into the system to solve the first issue. Specifically, to filter out distracting objects in the scene, we propose extracting face images of the speakers from the raw images and introduce a customized active face extraction strategy to rule out inactive speakers. As for the second issue, we employ efficient data transmission from the robot to the local server to improve realtime response capability. We deploy UGotMe on a human robot named Ameca to validate its real-time inference capabilities in practical scenarios. Videos demonstrating real-world deployment are available at https://pi3-141592653.github.io/UGotMe/.

Figures (5)

• Figure 1: In a multiparty human-robot conversation scenario, the active speaker (Tim) initiates the conversation by simply stating a fact "I am still working on my experiments" while holding a neutral face. The humanoid robot (Ameca) is supposed to deliver a neutral face in response to the speaker. However, two inactive speakers (Tom and Amy) and other persons (Jay and Ian) irrelevant to the conversation holding different facial expressions from the active speaker appear in the field of view of the robot, which may confuse the model, leading to the wrong answer.
• Figure 2: An overview of UGotMe, the proposed affective human-robot interaction system. The working pipeline includes on-robot multimodal perception (B) and on-edge vision-language to emotion modeling (C), where multimodal emotion recognition and robotic facial expression generation occur. D. Customized active face extraction (a)-(e) handle the environmental noise issue in Fig. \ref{['fig:distraction']}
• Figure 3: An illustration of the VL2E model.
• Figure 4: Two examples of real-world execution of UGotMe. Active speakers who are conversing with Ameca are indicated by the red bounding boxes.
• Figure 5: A comparison between UGotMe-TelME and UGotMe-VL2E. In both cases, the inactive speaker has a sad expression, while the active speaker, who is talking with Ameca, has a joyful expression. Dialogue context for both cases are: "The movie we saw last night is really impressive. That’s awesome. What movie did you watch? You jump, I jump". Ameca is supposed to deliver the same emotion as the active speaker through facial expression. However, in the case of UGotMe-TelME, distracting face confuses the model, leading to the wrong answer.