Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Test-Time Adaptation for Speech Emotion Recognition

Jiaheng Dong, Hong Jia, Ting Dang

TL;DR

This work addresses the challenge of speech emotion recognition (SER) under domain shifts by evaluating test-time adaptation (TTA) methods that use unlabeled target data. It categorizes TTA into entropy-minimization, pseudo-labeling, and backpropagation-free strategies, and systematically tests 11 methods across three SER tasks (intra-corpus personalization, acted-to-natural adaptation, and cross-corpus generalization) using a Wav2Vec 2.0 encoder. Key findings show that backpropagation-free methods are the most promising for SER, while entropy minimization and pseudo-labeling often underperform due to the intrinsic ambiguity of emotional labels; cross-corpus adaptation yields the largest gains, whereas acted-to-natural shifts are challenging for current TTA approaches. The results provide a practical foundation for designing SER systems that maintain performance in real-world, privacy-conscious scenarios and guide future directions toward more robust, task-sensitive TTA techniques.

Abstract

The practical utility of Speech Emotion Recognition (SER) systems is undermined by their fragility to domain shifts, such as speaker variability, the distinction between acted and naturalistic emotions, and cross-corpus variations. While domain adaptation and fine-tuning are widely studied, they require either source data or labelled target data, which are often unavailable or raise privacy concerns in SER. Test-time adaptation (TTA) bridges this gap by adapting models at inference using only unlabeled target data. Yet, having been predominantly designed for image classification and speech recognition, the efficacy of TTA for mitigating the unique domain shifts in SER has not been investigated. In this paper, we present the first systematic evaluation and comparison covering 11 TTA methods across three representative SER tasks. The results indicate that backpropagation-free TTA methods are the most promising. Conversely, entropy minimization and pseudo-labeling generally fail, as their core assumption of a single, confident ground-truth label is incompatible with the inherent ambiguity of emotional expression. Further, no single method universally excels, and its effectiveness is highly dependent on the distributional shifts and tasks.

Test-Time Adaptation for Speech Emotion Recognition

TL;DR

This work addresses the challenge of speech emotion recognition (SER) under domain shifts by evaluating test-time adaptation (TTA) methods that use unlabeled target data. It categorizes TTA into entropy-minimization, pseudo-labeling, and backpropagation-free strategies, and systematically tests 11 methods across three SER tasks (intra-corpus personalization, acted-to-natural adaptation, and cross-corpus generalization) using a Wav2Vec 2.0 encoder. Key findings show that backpropagation-free methods are the most promising for SER, while entropy minimization and pseudo-labeling often underperform due to the intrinsic ambiguity of emotional labels; cross-corpus adaptation yields the largest gains, whereas acted-to-natural shifts are challenging for current TTA approaches. The results provide a practical foundation for designing SER systems that maintain performance in real-world, privacy-conscious scenarios and guide future directions toward more robust, task-sensitive TTA techniques.

Abstract

The practical utility of Speech Emotion Recognition (SER) systems is undermined by their fragility to domain shifts, such as speaker variability, the distinction between acted and naturalistic emotions, and cross-corpus variations. While domain adaptation and fine-tuning are widely studied, they require either source data or labelled target data, which are often unavailable or raise privacy concerns in SER. Test-time adaptation (TTA) bridges this gap by adapting models at inference using only unlabeled target data. Yet, having been predominantly designed for image classification and speech recognition, the efficacy of TTA for mitigating the unique domain shifts in SER has not been investigated. In this paper, we present the first systematic evaluation and comparison covering 11 TTA methods across three representative SER tasks. The results indicate that backpropagation-free TTA methods are the most promising. Conversely, entropy minimization and pseudo-labeling generally fail, as their core assumption of a single, confident ground-truth label is incompatible with the inherent ambiguity of emotional expression. Further, no single method universally excels, and its effectiveness is highly dependent on the distributional shifts and tasks.
Paper Structure (17 sections, 6 equations, 2 figures, 1 table)

This paper contains 17 sections, 6 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Methodology
  3. Overview
  4. Adaptation via Entropy-minimization
  5. Adaptation via Pseudo-labeling
  6. Backpropagation-Free Methods
  7. Experimental setup
  8. Databases
  9. Source Model Structure and Training Settings
  10. Implementation Details of TTA
  11. Tasks
  12. Results
  13. Intra-corpus Personalization
  14. Performance on Acted-to-natural Adaptation
  15. Performance on Cross-corpus Adaptation
  16. ...and 2 more sections

Figures (2)

  • Figure 1: TTA evaluation for SER. The source model is adapted to the target dataset using only unlabeled speech data.
  • Figure 2: Category average accuracy with different batch size for Task 3 (a) IEMOCAP and (b) RAVDESS.