Exploring Self-Supervised Multi-view Contrastive Learning for Speech Emotion Recognition with Limited Annotations

TL;DR

This work tackles speech emotion recognition under limited annotations by introducing Pairwise-CL, a multi-view self-supervised pre-training framework that aligns representations across diverse speech views. It employs three views—wav2vec 2.0 features, mel spectrograms, and eGeMAPS-88—and optimizes a pairwise contrastive loss to maximize agreement among views of the same utterance while contrasting different utterances. In sparse-data regimes, Pairwise-CL yields significant gains in Unweighted Average Recall (UAR) over supervised baselines, with up to 10–15% improvements for handcrafted features and notable gains for other views, depending on data availability. Alignment analyses using PWCCA and visualization via t-SNE confirm improved cross-view coherence after pre-training. The approach offers a lightweight, flexible path to leverage large SSL speech models for SER in real-world, data-scarce settings and advocates incorporating additional views or modalities in future work.

Abstract

Recent advancements in Deep and Self-Supervised Learning (SSL) have led to substantial improvements in Speech Emotion Recognition (SER) performance, reaching unprecedented levels. However, obtaining sufficient amounts of accurately labeled data for training or fine-tuning the models remains a costly and challenging task. In this paper, we propose a multi-view SSL pre-training technique that can be applied to various representations of speech, including the ones generated by large speech models, to improve SER performance in scenarios where annotations are limited. Our experiments, based on wav2vec 2.0, spectral and paralinguistic features, demonstrate that the proposed framework boosts the SER performance, by up to 10% in Unweighted Average Recall, in settings with extremely sparse data annotations.

Figures (6)

• Figure 1: The proposed multi-view SSL framework for speech emotion recognition.
• Figure 2: Pairwise contrastive loss calculation. Representations from each view are first passed through a separate projection head. Later, the contrastive loss is computed in a pairwise fashion, according to Equations \ref{['eq:nt-xent']} - \ref{['eq:pairwise']}.
• Figure 3: UAR for fine-tuning with limited amounts of labeled data: * - statistically significant differences, ns - not significant.
• Figure 4: Comparison of model pre-trained on datasets with target (green) and out-of-distribution (red) annotations.
• Figure 5: Representations from the test set projected onto the two-dimensional space using t-SNE: (a)-(c) -- Pairwise-CL (before fine-tuning); (d)-(f) -- supervised training from scratch.