Awakening Facial Emotional Expressions in Human-Robot

TL;DR

This work tackles the challenge of natural facial expression generation in humanoid robots by introducing Rena, a biomimetic 25-DoF face, and an end-to-end learning framework based on Kolmogorov-Arnold Networks (KAN) with attention. An automated data-collection system yields a $9{,}000$-sample open-source dataset, enabling direct mapping from 52 blendshape coefficients to servomotor commands with real-time performance up to $50$ fps and low servo error ($4.4\%$). The approach demonstrates accurate, diverse facial mimicry across subjects and shows clear advantages over landmark-based and prior imitation-learning methods, supported by ablations and multiple datasets. The work contributes a high-fidelity animatronic platform, a low-parameter, fast regression model, and an open dataset to accelerate research in human-robot emotional interaction.

Abstract

The facial expression generation capability of humanoid social robots is critical for achieving natural and human-like interactions, playing a vital role in enhancing the fluidity of human-robot interactions and the accuracy of emotional expression. Currently, facial expression generation in humanoid social robots still relies on pre-programmed behavioral patterns, which are manually coded at high human and time costs. To enable humanoid robots to autonomously acquire generalized expressive capabilities, they need to develop the ability to learn human-like expressions through self-training. To address this challenge, we have designed a highly biomimetic robotic face with physical-electronic animated facial units and developed an end-to-end learning framework based on KAN (Kolmogorov-Arnold Network) and attention mechanisms. Unlike previous humanoid social robots, we have also meticulously designed an automated data collection system based on expert strategies of facial motion primitives to construct the dataset. Notably, to the best of our knowledge, this is the first open-source facial dataset for humanoid social robots. Comprehensive evaluations indicate that our approach achieves accurate and diverse facial mimicry across different test subjects.

Figures (7)

• Figure 1: Rena is a general animatronic robotic face for emotional expressions. The robot achieves this by learning the correspondence between its facial feature representations and the control of servos. The entire learning process relies on the robot issuing motor commands based on specific expert strategies. The figure illustrates the robot's capability to replicate a variety of human expressions, with a particular emphasis on the realism of these expressions.
• Figure 2: The exploded view of the Rena robot is shown in (b), where (c) is a cross-sectional view of the robot structure, showing the 12 DoF motion trajectory of the mouth motion unit and the 8 DoF motion trajectory of the eyebrow area. (a) shows our bionic silicone skin and bionic eyeball.
• Figure 3: Our method consists of two modules: dataset construction and network training. The dataset construction module automatically generates commands corresponding to random expressions using an expert strategy system, followed by capturing images with a conventional RGB camera. The network training module extracts facial representation blendshape coefficients using MediaPipe. Subsequently, our designed network performs regression to fit the servo commands.
• Figure 4: A visual representation of how different facial regions contribute differently to the driving weights of different servos.
• Figure 5: We performed output servomotor commands on our Rena robot on both the MMI dataset and the open dataset to demonstrate that our method supports accurate simulation of various human expressions in multiple human subjects.