Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance

Huang-Cheng Chou, Haibin Wu, Hung-yi Lee, Chi-Chun Lee

TL;DR

The paper addresses how the modality used to elicit emotional labels influences SER performance, comparing audio-only, face-only, audio-visual, and all-inclusive labels using SSLMs. It introduces an all-inclusive label set that fuses labels from multiple modalities and evaluates models under cross-modal testing conditions. Key findings show that voice-only labeling yields the best results for speech-only testing, while the all-inclusive label set enhances performance on face-only and audio-visual test conditions, highlighting modality-dependent labeling effects. The work advances practical SER design by aligning labeling strategies with the intended testing scenario and suggests extending approaches to multi-modal input systems in the future.

Abstract

Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in different ways. For instance, the IEMOCAP dataset uses video clips with sounds for annotators to provide their emotional perceptions. However, the most significant English emotion dataset, the MSP-PODCAST, only provides speech for raters to choose the emotional ratings. Nevertheless, using speech as input is the standard approach to training SER systems. Therefore, the open question is the emotional labels elicited by which scenarios are the most effective for training SER systems. We comprehensively compare the effectiveness of SER systems trained with labels elicited by different modality stimuli and evaluate the SER systems on various testing conditions. Also, we introduce an all-inclusive label that combines all labels elicited by various modalities. We show that using labels elicited by voice-only stimuli for training yields better performance on the test set, whereas labels elicited by voice-only stimuli.

Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance

TL;DR

The paper addresses how the modality used to elicit emotional labels influences SER performance, comparing audio-only, face-only, audio-visual, and all-inclusive labels using SSLMs. It introduces an all-inclusive label set that fuses labels from multiple modalities and evaluates models under cross-modal testing conditions. Key findings show that voice-only labeling yields the best results for speech-only testing, while the all-inclusive label set enhances performance on face-only and audio-visual test conditions, highlighting modality-dependent labeling effects. The work advances practical SER design by aligning labeling strategies with the intended testing scenario and suggests extending approaches to multi-modal input systems in the future.

Abstract

Speech Emotion Recognition (SER) systems rely on speech input and emotional labels annotated by humans. However, various emotion databases collect perceptional evaluations in different ways. For instance, the IEMOCAP dataset uses video clips with sounds for annotators to provide their emotional perceptions. However, the most significant English emotion dataset, the MSP-PODCAST, only provides speech for raters to choose the emotional ratings. Nevertheless, using speech as input is the standard approach to training SER systems. Therefore, the open question is the emotional labels elicited by which scenarios are the most effective for training SER systems. We comprehensively compare the effectiveness of SER systems trained with labels elicited by different modality stimuli and evaluate the SER systems on various testing conditions. Also, we introduce an all-inclusive label that combines all labels elicited by various modalities. We show that using labels elicited by voice-only stimuli for training yields better performance on the test set, whereas labels elicited by voice-only stimuli.
Paper Structure (14 sections, 2 figures, 4 tables)

This paper contains 14 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Methodology
  4. Labels Elicited by Multi-modal Emotional Stimulus
  5. Proposed All-inclusive Label Set and Rationales
  6. SSLMs-based SER Framework
  7. Experimental Settings
  8. The CREAMA-D
  9. Class-balanced Objective Function
  10. Evaluation Metrics and Confidence Intervals
  11. Models Training and Choice
  12. Results and Analysis
  13. Discussion and Limitations
  14. Conclusion and Future Work

Figures (2)

  • Figure 1: The figure shows the multi-modal emotional stimulus in the CREMA-D emotion corpus and the proposed all-inclusive label.
  • Figure 2: The layerwise weights of WavLM-based SER systems trained with the labels elicited by various modalities. The Face, Voice, and AV represent face-only, voice-only, audio-visual, and all-inclusive, respectively.