Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enrolment-based personalisation for improving individual-level fairness in speech emotion recognition

Andreas Triantafyllopoulos, Björn Schuller

TL;DR

The paper tackles fairness in speech emotion recognition (SER) by moving beyond population-level metrics to per-speaker evaluations. It introduces an enrolment-based personalisation framework that uses a two-encoder attention mechanism to adapt SER models to unseen speakers with a minimal set of enrolment utterances. The authors also formalise individual-level fairness through speaker-level utility $UAR_{SP}$, the Gini coefficient, and ISWF, and demonstrate that enrolment-based adaptation can improve global performance and fairness on FAU-AIBO and MSP-Podcast. The work highlights practical implications for fair SER deployment and points to future directions in explainability and robustness against erroneous enrolment data.

Abstract

The expression of emotion is highly individualistic. However, contemporary speech emotion recognition (SER) systems typically rely on population-level models that adopt a `one-size-fits-all' approach for predicting emotion. Moreover, standard evaluation practices measure performance also on the population level, thus failing to characterise how models work across different speakers. In the present contribution, we present a new method for capitalising on individual differences to adapt an SER model to each new speaker using a minimal set of enrolment utterances. In addition, we present novel evaluation schemes for measuring fairness across different speakers. Our findings show that aggregated evaluation metrics may obfuscate fairness issues on the individual-level, which are uncovered by our evaluation, and that our proposed method can improve performance both in aggregated and disaggregated terms.

Enrolment-based personalisation for improving individual-level fairness in speech emotion recognition

TL;DR

The paper tackles fairness in speech emotion recognition (SER) by moving beyond population-level metrics to per-speaker evaluations. It introduces an enrolment-based personalisation framework that uses a two-encoder attention mechanism to adapt SER models to unseen speakers with a minimal set of enrolment utterances. The authors also formalise individual-level fairness through speaker-level utility , the Gini coefficient, and ISWF, and demonstrate that enrolment-based adaptation can improve global performance and fairness on FAU-AIBO and MSP-Podcast. The work highlights practical implications for fair SER deployment and points to future directions in explainability and robustness against erroneous enrolment data.

Abstract

The expression of emotion is highly individualistic. However, contemporary speech emotion recognition (SER) systems typically rely on population-level models that adopt a `one-size-fits-all' approach for predicting emotion. Moreover, standard evaluation practices measure performance also on the population level, thus failing to characterise how models work across different speakers. In the present contribution, we present a new method for capitalising on individual differences to adapt an SER model to each new speaker using a minimal set of enrolment utterances. In addition, we present novel evaluation schemes for measuring fairness across different speakers. Our findings show that aggregated evaluation metrics may obfuscate fairness issues on the individual-level, which are uncovered by our evaluation, and that our proposed method can improve performance both in aggregated and disaggregated terms.
Paper Structure (10 sections, 2 equations, 2 figures, 1 table)

This paper contains 10 sections, 2 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Datasets
  4. Personalisation via enrolment
  5. Experimental settings
  6. Individual-level fairness
  7. Results & Discussion
  8. Conclusion
  9. Acknowledgements

Figures (2)

  • Figure 1: Overview of the proposed architecture. A set of enrolment utterances is passed through an encoder to generate enrolment embeddings. These are used as keys and values in a dot-product attention scheme with the embeddings generated from the target utterance. The output embedding is passed to a feed-forward neural network for the final classification. Weights are shared between the enrolment and the main encoders.
  • Figure 2: Total utility achieved by each model for different isoelastic social welfare functions for the $2$- (top) and $5$-class (bottom) formulations of FAU-AIBO. Utility is defined as $UAR_{SP}$.