Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

Lucas H. Ueda, João G. T. Lima, Paula D. P. Costa

Abstract

Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.

Crab: Multi Layer Contrastive Supervision to Improve Speech Emotion Recognition Under Both Acted and Natural Speech Condition

Abstract

Speech Emotion Recognition (SER) in real-world scenarios remains challenging due to severe class imbalance and the prevalence of spontaneous, natural speech. While recent approaches leverage self-supervised learning (SSL) representations and multimodal fusion of speech and text, most existing methods apply supervision only at the final classification layer, limiting the discriminative power of intermediate representations. In this work, we propose Crab (Contrastive Representation and Multimodal Aligned Bottleneck), a bimodal Cross-Modal Transformer architecture that integrates speech representations from WavLM and textual representations from RoBERTa, together with a novel \textit{Multi Layer Contrastive Supervision} (MLCS) strategy. MLCS injects multi-positive contrastive learning signals at multiple layers of the network, encouraging emotionally discriminative representations throughout the model without introducing additional parameters at inference time. To further address data imbalance, we adopt weighted cross-entropy during training. We evaluate the proposed approach on three benchmark datasets covering different degrees of emotional naturalness: IEMOCAP, MELD, and MSP-Podcast 2.0. Experimental results demonstrate that Crab consistently outperforms strong unimodal and multimodal baselines across all datasets, with particularly large gains under naturalistic and highly imbalanced conditions. These findings highlight the effectiveness of \textit{Multi Layer Contrastive Supervision} as a general and robust strategy for SER. Official implementation can be found in https://github.com/AI-Unicamp/Crab.
Paper Structure (19 sections, 9 equations, 3 figures, 8 tables)

This paper contains 19 sections, 9 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Multimodality
  4. Contrastive Learning
  5. Deep Supervision
  6. Methodology
  7. Datasets
  8. Crab Model
  9. Cross-Modal Transformer
  10. Multi Layer Contrastive Supervision (MLCS)
  11. Training Design
  12. Evaluation Metrics
  13. Results
  14. MLCS weight ($\alpha$) in the loss
  15. Multimodality
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Crab model architecture. The modules highlighted in green in each block have the Contrastive Guidance Leg attached and are optimized using the proposed MPCL contrastive loss.
  • Figure 2: Confusion matrices for MELD dataset across all evaluated models. Values in red on the principal diagonal represent values under a threshold of 0.2. Note that Crab has higher values on the principal diagonal compared to baselines with no emotion category having values under the 0.2 treshold. In particular, the proposed model is robust for the least represented classes (Fear, Sadness, and Disgust).
  • Figure 3: UAR $\times$ WAR performance for different values of $\alpha$. The dashed horizontal lines indicate the proposed model trained only with CE and the Medusa model, using UAR as a reference.