Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation

Nineli Lashkarashvili, Wen Wu, Guangzhi Sun, Philip C. Woodland

TL;DR

The paper tackles the data scarcity challenge in speech emotion recognition by evaluating parameter-efficient finetuning (PEFT) methods on both discrete emotion categorization and dimensional attribute prediction. It introduces a two-stage domain adaptation framework that starts from acted emotion data and progressively adapts to natural emotion using PEFT adapters, with intra- and cross-corpus validation. The results show that combinations of Bottleneck, LoRA, Weighted Sum, and Weight-Gating adapters achieve comparable or superior performance to full finetuning while updating only a small fraction of parameters, and the proposed acted-to-natural adaptation improves robustness across domains. This work provides a practical pathway for deploying SER systems with strong performance under limited data and cross-domain variability, without relying on external features beyond raw audio.

Abstract

Foundation models have shown superior performance for speech emotion recognition (SER). However, given the limited data in emotion corpora, finetuning all parameters of large pre-trained models for SER can be both resource-intensive and susceptible to overfitting. This paper investigates parameter-efficient finetuning (PEFT) for SER. Various PEFT adaptors are systematically studied for both classification of discrete emotion categories and prediction of dimensional emotional attributes. The results demonstrate that the combination of PEFT methods surpasses full finetuning with a significant reduction in the number of trainable parameters. Furthermore, a two-stage adaptation strategy is proposed to adapt models trained on acted emotion data, which is more readily available, to make the model more adept at capturing natural emotional expressions. Both intra- and cross-corpus experiments validate the efficacy of the proposed approach in enhancing the performance on both the source and target domains.

Parameter Efficient Finetuning for Speech Emotion Recognition and Domain Adaptation

TL;DR

The paper tackles the data scarcity challenge in speech emotion recognition by evaluating parameter-efficient finetuning (PEFT) methods on both discrete emotion categorization and dimensional attribute prediction. It introduces a two-stage domain adaptation framework that starts from acted emotion data and progressively adapts to natural emotion using PEFT adapters, with intra- and cross-corpus validation. The results show that combinations of Bottleneck, LoRA, Weighted Sum, and Weight-Gating adapters achieve comparable or superior performance to full finetuning while updating only a small fraction of parameters, and the proposed acted-to-natural adaptation improves robustness across domains. This work provides a practical pathway for deploying SER systems with strong performance under limited data and cross-domain variability, without relying on external features beyond raw audio.

Abstract

Foundation models have shown superior performance for speech emotion recognition (SER). However, given the limited data in emotion corpora, finetuning all parameters of large pre-trained models for SER can be both resource-intensive and susceptible to overfitting. This paper investigates parameter-efficient finetuning (PEFT) for SER. Various PEFT adaptors are systematically studied for both classification of discrete emotion categories and prediction of dimensional emotional attributes. The results demonstrate that the combination of PEFT methods surpasses full finetuning with a significant reduction in the number of trainable parameters. Furthermore, a two-stage adaptation strategy is proposed to adapt models trained on acted emotion data, which is more readily available, to make the model more adept at capturing natural emotional expressions. Both intra- and cross-corpus experiments validate the efficacy of the proposed approach in enhancing the performance on both the source and target domains.
Paper Structure (14 sections, 3 equations, 2 figures, 3 tables)

This paper contains 14 sections, 3 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Model structure
  4. PEFT adaptors
  5. Domain adaptation from acted emotion to natural emotion
  6. Experimental setup
  7. Datasets
  8. Implementation details
  9. Results
  10. Efficient finetuning for SER
  11. Adaptation from acted emotion to natural emotion
  12. Intra-corpus adaptation
  13. Cross-corpus adaptation
  14. Conclusion

Figures (2)

  • Figure 1: A wav2vec 2.0 / HuBERT Transformer layer with various PEFT adaptors
  • Figure 2: Proposed two-stage domain adaptation from acted to natural emotion prediction. denotes that PEFT module is updated while that it is frozen. The figure depicts the scenario when BA and LoRA weights from stage 1 are not updated.