Improving Inclusivity for Emotion Recognition Based on Face Tracking

TL;DR

This paper tackles inclusive emotion recognition in MR/VR via face tracking, highlighting how individual, cultural, and neurodiverse differences challenge accuracy. It proposes a multi-faceted approach: expanding diverse datasets (acted and natural emotions, varied intensities, complex expressions, neurodiverse participants), leveraging contextual multimodal cues (audio, text, and peer states), and implementing transparent, user-centered calibration and control mechanisms. Key contributions include guidance on dataset design, context-aware inference, and interactive calibration, with mechanisms for users to adjust or trigger emotions. It also foregrounds privacy and ethical considerations, arguing for systems that adapt to users while granting control over data usage and emphasizing avoidance of pressuring users to alter their expressive style.

Abstract

The limited expressiveness of virtual user representations in Mixed Reality and Virtual Reality can inhibit an integral part of communication: emotional expression. Emotion recognition based on face tracking is often used to compensate for this. However, emotional facial expressions are highly individual, which is why many approaches have difficulties recognizing unique variations of emotional expressions. We propose several strategies to improve face tracking systems for emotion recognition with and without user intervention for the Affective Interaction Workshop at CHI '25.