Explainable Transformer-CNN Fusion for Noise-Robust Speech Emotion Recognition

TL;DR

This work targets noise-robust Speech Emotion Recognition by fusing a Transformer-based temporal model (Wav2Vec 2.0) with a spectral-stability CNN, guided by Attentive Temporal Pooling. A comprehensive data strategy across four corpora and SAS-KIIT non-stationary noise demonstrates strong cross-corpus generalization and resilience to real-world interference. The study also integrates SHAP and Score-CAM to provide granular explanations, showing the model emphasizes linguistically meaningful spectral and prosodic cues rather than background noise. Ablation analyses quantify the necessity of the dual-stream fusion and attentive pooling, while XAI analyses confirm that predictions align with interpretable acoustic features, supporting deployment in trustworthy SER applications.

Abstract

Speech Emotion Recognition (SER) systems often degrade in performance when exposed to the unpredictable acoustic interference found in real-world environments. Additionally, the opacity of deep learning models hinders their adoption in trust-sensitive applications. To bridge this gap, we propose a Hybrid Transformer-CNN framework that unifies the contextual modeling of Wav2Vec 2.0 with the spectral stability of 1D-Convolutional Neural Networks. Our dual-stream architecture processes raw waveforms to capture long-range temporal dependencies while simultaneously extracting noise-resistant spectral features (MFCC, ZCR, RMSE) via a custom Attentive Temporal Pooling mechanism. We conducted extensive validation across four diverse benchmark datasets: RAVDESS, TESS, SAVEE, and CREMA-D. To rigorously test robustness, we subjected the model to non-stationary acoustic interference using real-world noise profiles from the SAS-KIIT dataset. The proposed framework demonstrates superior generalization and state-of-the-art accuracy across all datasets, significantly outperforming single-branch baselines under realistic environmental interference. Furthermore, we address the ``black-box" problem by integrating SHAP and Score-CAM into the evaluation pipeline. These tools provide granular visual explanations, revealing how the model strategically shifts attention between temporal and spectral cues to maintain reliability in the presence of complex environmental noise.

Figures (5)

• Figure 1: The proposed architecture combining a pre-trained Transformer backbone with a dedicated CNN feature encoder. Branch I utilizes Wav2Vec with a learnable attention mechanism to derive contextual embeddings, while Branch II extracts robust spectral embeddings from MFCC, RMSE, and ZCR inputs.
• Figure 2: Score-CAM Saliency Visualization. The figure displays heatmaps for (a-b) Neutral, (c-d) Angry, and (e-f) Disgust emotions. In the left panels, the waveform is overlaid with a heatmap where red/yellow regions denote high importance (positive contribution to the prediction) and blue/purple regions denote low importance. The right panels show the corresponding discrete saliency intensity per time step.
• Figure 3: Signal Alignment and Causal Perturbation Testing.(a-b) Dual-axis plots showing the alignment of importance scores with the raw waveform for 'Angry' and 'Disgust'. (c) Occlusion sensitivity analysis for the 'Fear' class. (d) Visualization of active vs. ignored regions for 'Surprise'. (e) Counterfactual perturbation test comparing the original signal (Plot X) against a masked signal where high-importance regions are removed (Plot Y).
• Figure 4: SHAP Feature Contribution Analysis. This multi-panel figure quantifies the impact of specific temporal features on the model's predictions. Panel (a) ranks the top features by global importance across all emotions. Panels (b) and (c) visualize the distribution of feature impacts specifically for the 'fear' class. Panel (d) provides a local breakdown, showing how individual feature contributions accumulate to form a single prediction score.
• Figure 5: Temporal SHAP Attribution ("The Glowing Signal"). These scatter plots project SHAP importance values directly onto the audio waveform for (a) Fear, (b-c) Surprise, and (d) Happy. Each dot represents a time-step; the color intensity indicates the magnitude of the contribution toward the target emotion prediction (Warmer/Darker colors = Strong Positive Impact).