Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Power of Properties: Uncovering the Influential Factors in Emotion Classification

Tim Büchner, Niklas Penzel, Orlando Guntinas-Lichius, Joachim Denzler

TL;DR

The paper addresses the opacity of end-to-end emotion classifiers by introducing a causal-analysis framework that treats observable facial properties (e.g., age, gender, facial symmetry, sEMG) as factors affecting model decisions. Using a structural causal model and a committee of nonlinear conditional-independence tests, it evaluates two state-of-the-art models (HSEmotion-7 and RMN) on a medically sourced dataset with probands and facial-palsy patients, measuring logit changes across Ekman emotions. It reports that up to $91.25\%$ of property manifestations induce significant changes in model outputs ($p<0.01$) and highlights effects from gender, sEMG, and facial symmetry on predictions. The findings underscore the importance of property-aware evaluation for medical deployments and motivate broader, causally grounded analyses beyond mere accuracy metrics.

Abstract

Facial expression-based human emotion recognition is a critical research area in psychology and medicine. State-of-the-art classification performance is only reached by end-to-end trained neural networks. Nevertheless, such black-box models lack transparency in their decision-making processes, prompting efforts to ascertain the rules that underlie classifiers' decisions. Analyzing single inputs alone fails to expose systematic learned biases. These biases can be characterized as facial properties summarizing abstract information like age or medical conditions. Therefore, understanding a model's prediction behavior requires an analysis rooted in causality along such selected properties. We demonstrate that up to 91.25% of classifier output behavior changes are statistically significant concerning basic properties. Among those are age, gender, and facial symmetry. Furthermore, the medical usage of surface electromyography significantly influences emotion prediction. We introduce a workflow to evaluate explicit properties and their impact. These insights might help medical professionals select and apply classifiers regarding their specialized data and properties.

The Power of Properties: Uncovering the Influential Factors in Emotion Classification

TL;DR

The paper addresses the opacity of end-to-end emotion classifiers by introducing a causal-analysis framework that treats observable facial properties (e.g., age, gender, facial symmetry, sEMG) as factors affecting model decisions. Using a structural causal model and a committee of nonlinear conditional-independence tests, it evaluates two state-of-the-art models (HSEmotion-7 and RMN) on a medically sourced dataset with probands and facial-palsy patients, measuring logit changes across Ekman emotions. It reports that up to of property manifestations induce significant changes in model outputs () and highlights effects from gender, sEMG, and facial symmetry on predictions. The findings underscore the importance of property-aware evaluation for medical deployments and motivate broader, causally grounded analyses beyond mere accuracy metrics.

Abstract

Facial expression-based human emotion recognition is a critical research area in psychology and medicine. State-of-the-art classification performance is only reached by end-to-end trained neural networks. Nevertheless, such black-box models lack transparency in their decision-making processes, prompting efforts to ascertain the rules that underlie classifiers' decisions. Analyzing single inputs alone fails to expose systematic learned biases. These biases can be characterized as facial properties summarizing abstract information like age or medical conditions. Therefore, understanding a model's prediction behavior requires an analysis rooted in causality along such selected properties. We demonstrate that up to 91.25% of classifier output behavior changes are statistically significant concerning basic properties. Among those are age, gender, and facial symmetry. Furthermore, the medical usage of surface electromyography significantly influences emotion prediction. We introduce a workflow to evaluate explicit properties and their impact. These insights might help medical professionals select and apply classifiers regarding their specialized data and properties.
Paper Structure (9 sections, 1 figure, 3 tables)

This paper contains 9 sections, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Facial Emotion Classification
  3. Emotion Evaluation Dataset
  4. Facial Properties: Selection and Manifestations
  5. Methodology
  6. Results
  7. Conclusion
  8. Acknowledgments.
  9. Disclosure of Interests.

Figures (1)

  • Figure 1: Model behavior visualization: In the binary [B] case, we use violin plots for logit distributions (\ref{['fig:analysis_gender']}, \ref{['fig:analysis_emg']}, \ref{['fig:analysis_fcp']}). For the scalar [S] property$\mathfrak{S}_L$, we plot a regression analysis by estimating Gaussians with a sliding window penzel2023analyzing. For all visualizations, we only use images of the corresponding emotion reimers2020determining.