Multi-face emotion detection for effective Human-Robot Interaction

TL;DR

The paper tackles real-time multi-face emotion detection for human-robot interaction on resource-constrained mobile humanoid robots. It conducts a comparative study of transfer-learned FER models trained on FER2013 and selects EfficientNetV2-B0 for deployment on the Tiago++ robot due to its favorable accuracy-to-footprint balance, while employing Haarcascade for multi-face detection and a Tkinter/OpenCV GUI for visualization. The system is integrated with ROS to enable on-board face tracking and emotion-driven GUI display on the robot’s tablet, demonstrating real-time performance with single and multi-person scenarios within a ~139 MB model budget. The work highlights the practical viability of compact FER on robots while acknowledging the limitations of facial cues alone and proposing multimodal extensions and further optimization for robust HRI applications.

Abstract

The integration of dialogue interfaces in mobile devices has become ubiquitous, providing a wide array of services. As technology progresses, humanoid robots designed with human-like features to interact effectively with people are gaining prominence, and the use of advanced human-robot dialogue interfaces is continually expanding. In this context, emotion recognition plays a crucial role in enhancing human-robot interaction by enabling robots to understand human intentions. This research proposes a facial emotion detection interface integrated into a mobile humanoid robot, capable of displaying real-time emotions from multiple individuals on a user interface. To this end, various deep neural network models for facial expression recognition were developed and evaluated under consistent computer-based conditions, yielding promising results. Afterwards, a trade-off between accuracy and memory footprint was carefully considered to effectively implement this application on a mobile humanoid robot.

Figures (9)

• Figure 1: Cascade structure for Haar classifiers kim2015system.
• Figure 2: The architecture of the emotion recognition system using transfer learning on the FER2013 dataset.
• Figure 3: Global architecture of our real-time multi-face emotion recognition user interface.
• Figure 4: ROS-based Tiago++ face emotion recognition integration process: the diagram in the left (a) depicts the steps involved in face tracking integration, while the diagram in the right (b) shows the emotion detection and GUI display process.
• Figure 5: Accuracy and confidence intervals of the models