"I think you need help! Here's why": Understanding the Effect of Explanations on Automatic Facial Expression Recognition

TL;DR

This study tackles the challenge of transparency in facial expression recognition (FER) by applying intrinsic FAU-based explanations within an XAI framework to an FER-driven hint system in a navigation game. Using a between-subject design, it compares explanations (global and local) against a control of hints without explanations, showing that explanations improve user understanding, increase hint acceptance, reduce collisions, and elevate trust. Key contributions include an open, modular framework for testing FER explainability in HCI, empirical evidence of its benefits, and a dataset with interaction, survey, and interview data for replication. The findings support the practical value of explanations in emotion-aware systems, while highlighting the need for careful management of trust and future extension to deep-learning FER models.

Abstract

Facial expression recognition (FER) has emerged as a promising approach to the development of emotion-aware intelligent systems. The performance of FER in multiple domains is continuously being improved, especially through advancements in data-driven learning approaches. However, a key challenge remains in utilizing FER in real-world contexts, namely ensuring user understanding of these systems and establishing a suitable level of user trust towards this technology. We conducted an empirical user study to investigate how explanations of FER can improve trust, understanding and performance in a human-computer interaction task that uses FER to trigger helpful hints during a navigation game. Our results showed that users provided with explanations of the FER system demonstrated improved control in using the system to their advantage, leading to a significant improvement in their understanding of the system, reduced collisions in the navigation game, as well as increased trust towards the system.

Figures (6)

• Figure 1: The Facial Expression Recognition System triggers hints based on a user's facial expressions.
• Figure 2: Local Explainer User Interface. Users can review the predicted emotion category (top of the window), the corresponding frames with facial landmark annotations (top panel, face anonymised), and facial action unit activation information (left panel).
• Figure 3: Framework Overview. 1) The user interacts with a computer, while their face is visible to a camera. 2) The system provides hints in the game (pop-up) based on the user's facial expressions. 3a) Users from the control cohort are asked if they would like a hint before being provided with one. 3b) Users from the explanation cohort view the same but are also given the option to view explanations on the FER system's output. Users from both cohorts have the opportunity to accept or reject the hints triggered by their facial expression.
• Figure 4: Explorer Game Interface (a) and Hint Trigger Mechanism Interface (b).
• Figure 5: Boxplot on collisions during Explorer Game. Significantly less collisions for XAutoHint group compared to AutoHint group