Recovering Performance in Speech Emotion Recognition from Discrete Tokens via Multi-Layer Fusion and Paralinguistic Feature Integration

TL;DR

The paper addresses information loss when converting continuous SSL representations into discrete tokens for SER. It introduces a framework that combines layer-wise discrete tokens from a fine-tuned WavLM-Large with attention-based cross-layer fusion and OpenSMILE-derived paralinguistic features, and it compares neural codecs as alternative tokenizers. Key findings show that multi-layer fusion recovers a substantial portion of the discretization gap (about 62–75%), and OpenSMILE features can reintroduce paralinguistic cues to boost performance, while SSL-based tokens outperform neural codecs for SER. Together, these results demonstrate that discrete representations can achieve near-continuous SER performance when paired with structured fusion and acoustic augmentation, enabling efficient token-based SER pipelines.

Abstract

Discrete speech tokens offer significant advantages for storage and language model integration, but their application in speech emotion recognition (SER) is limited by paralinguistic information loss during quantization. This paper presents a comprehensive investigation of discrete tokens for SER. Using a fine-tuned WavLM-Large model, we systematically quantify performance degradation across different layer configurations and k-means quantization granularities. To recover the information loss, we propose two key strategies: (1) attention-based multi-layer fusion to recapture complementary information from different layers, and (2) integration of openSMILE features to explicitly reintroduce paralinguistic cues. We also compare mainstream neural codec tokenizers (SpeechTokenizer, DAC, EnCodec) and analyze their behaviors when fused with acoustic features. Our findings demonstrate that through multi-layer fusion and acoustic feature integration, discrete tokens can close the performance gap with continuous representations in SER tasks.

Figures (4)

• Figure 1: Proposed discrete SER framework. (a) Multi-layer discrete units from fine-tuned WavLM with layer-specific codebooks. (b) Neural codec tokenization via EnCodec, DAC, and SpeechTokenizer. (c) Hierarchical fusion augmenting discrete speech representations with quantized OpenSMILE paralinguistic features. All models share the same downstream architecture (layer attention, pooling, classifier).
• Figure 2: Performance (Macro F1 Score) across different WavLM layer configurations and codebook sizes (K). The continuous feature-based model (red line) is included as a reference baseline, using the corresponding layers.
• Figure 3: Analysis of attention weights across all 24 WavLM layers, $K = 1{,}000$, Macro F1 = 0.3441. The distribution is notably bimodal: the final two layers (L22, L23) are critically important, while select early layers also make contributions.
• Figure 4: Performance improvement (%) from augmenting discrete WavLM models ($K = 1{,}000$) with paralinguistic features. The results show a clear inverse relationship: sparser layer configurations (left) benefit the most from explicit paralinguistic cues, while denser configurations that already contain richer information (right) show diminishing returns.