Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition

Shreya G. Upadhyay, Carlos Busso, Chi-Chun Lee

TL;DR

The paper tackles cross-lingual speech emotion recognition by exploiting information across transformer layers rather than solely the final layer. It introduces a layer anchoring mechanism (LAM) that uses a CORAL loss to align layer representations across languages, guided by layer similarity analysis on MSP-Podcast and BIIC-Podcast using WavLM and Whisper. The proposed LAM-GL approach with group-layer anchors achieves a new best UAR of 60.21% on MSP-P→BIIC-P, outperforming baselines, and vowel-based layer selection further enhances cross-lingual transfer in some configurations. The work highlights that layer selection is model- and task-dependent and provides a practical framework for improving cross-lingual SER with pretrained transformers.

Abstract

Cross-lingual speech emotion recognition (SER) is important for a wide range of everyday applications. While recent SER research relies heavily on large pretrained models for emotion training, existing studies often concentrate solely on the final transformer layer of these models. However, given the task-specific nature and hierarchical architecture of these models, each transformer layer encapsulates different levels of information. Leveraging this hierarchical structure, our study focuses on the information embedded across different layers. Through an examination of layer feature similarity across different languages, we propose a novel strategy called a layer-anchoring mechanism to facilitate emotion transfer in cross-lingual SER tasks. Our approach is evaluated using two distinct language affective corpora (MSP-Podcast and BIIC-Podcast), achieving a best UAR performance of 60.21% on the BIIC-podcast corpus. The analysis uncovers interesting insights into the behavior of popular pretrained models.

A Layer-Anchoring Strategy for Enhancing Cross-Lingual Speech Emotion Recognition

TL;DR

The paper tackles cross-lingual speech emotion recognition by exploiting information across transformer layers rather than solely the final layer. It introduces a layer anchoring mechanism (LAM) that uses a CORAL loss to align layer representations across languages, guided by layer similarity analysis on MSP-Podcast and BIIC-Podcast using WavLM and Whisper. The proposed LAM-GL approach with group-layer anchors achieves a new best UAR of 60.21% on MSP-P→BIIC-P, outperforming baselines, and vowel-based layer selection further enhances cross-lingual transfer in some configurations. The work highlights that layer selection is model- and task-dependent and provides a practical framework for improving cross-lingual SER with pretrained transformers.

Abstract

Cross-lingual speech emotion recognition (SER) is important for a wide range of everyday applications. While recent SER research relies heavily on large pretrained models for emotion training, existing studies often concentrate solely on the final transformer layer of these models. However, given the task-specific nature and hierarchical architecture of these models, each transformer layer encapsulates different levels of information. Leveraging this hierarchical structure, our study focuses on the information embedded across different layers. Through an examination of layer feature similarity across different languages, we propose a novel strategy called a layer-anchoring mechanism to facilitate emotion transfer in cross-lingual SER tasks. Our approach is evaluated using two distinct language affective corpora (MSP-Podcast and BIIC-Podcast), achieving a best UAR performance of 60.21% on the BIIC-podcast corpus. The analysis uncovers interesting insights into the behavior of popular pretrained models.
Paper Structure (11 sections, 4 equations, 2 figures, 3 tables)

This paper contains 11 sections, 4 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Layer Similarity Analysis
  3. Naturalistic Corpora
  4. Layer Similarities
  5. Unified Layer Selection
  6. Layer Anchored Cross-Lingual SER
  7. Experiment Results and Analyses
  8. Performance Comparison
  9. CL-SER With Different Layer Selection Strategy
  10. Conclusion
  11. Acknowledgements

Figures (2)

  • Figure 1: Phonetic similarities across feature layer representations in MSP-P and BIIC-P corpora for WavLM and Whisper models, illustrating vowel-level and utterance-level phonetic similarities.
  • Figure 2: Proposed contrastive learning approach using layer anchoring mechanism (LAM) for cross-lingual SER.