Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mobile-Efficient Speech Emotion Recognition Using DistilHuBERT: A Cross-Corpus Validation Study

Saifelden M. Ismail

TL;DR

This work targets mobile-friendly Speech Emotion Recognition by deploying DistilHuBERT, a compact, 8-bit quantized transformer, achieving 23 MB and about 91% of full-scale Wav2Vec 2.0 performance under a strict 5-fold LOSO evaluation on IEMOCAP. It introduces cross-corpus regularization with CREMA-D to improve generalization, yielding a 1.2% gain in Weighted Accuracy and a 32% reduction in cross-fold variance, with notable Neutral class improvement but some Sadness recall decline due to a theatricality gap. Cross-corpus testing on RAVDESS reveals arousal-focused clustering of errors rather than random misclassification, with strong arousal detection (Anger recall 97%, Sadness 64%), illustrating domain shift effects. The paper also proposes a practical deployment architecture leveraging VAD, windowed processing, and temporal pooling to enable robust, privacy-preserving on-device SER on resource-constrained mobile devices.

Abstract

Speech Emotion Recognition (SER) has significant potential for mobile applications, yet deployment remains constrained by the computational demands of state-of-the-art transformer architectures. This paper presents a mobile-efficient SER system based on DistilHuBERT, a distilled and 8-bit quantized transformer that achieves 92% parameter reduction compared to full-scale Wav2Vec 2.0 models while maintaining competitive accuracy. We conduct a rigorous 5-fold Leave-One-Session-Out (LOSO) cross-validation on the IEMOCAP dataset to ensure speaker independence, augmented with cross-corpus training on CREMA-D to enhance generalization. Cross-corpus training with CREMA-D yields a 1.2% improvement in Weighted Accuracy, a 1.4% gain in Macro F1-score, and a 32% reduction in cross-fold variance, with the Neutral class showing the most substantial benefit at 5.4% F1-score improvement. Our approach achieves an Unweighted Accuracy of 61.4% with a quantized model footprint of only 23 MB, representing approximately 91% of full-scale baseline performance. Cross-corpus evaluation on RAVDESS reveals that the theatrical nature of acted emotions causes predictions to cluster by arousal level rather than valence: happiness is systematically confused with anger due to acoustic saturation in high-energy expressions. Despite this theatricality effect reducing overall RAVDESS accuracy to 43.29%, the model maintains robust arousal detection with 97% recall for anger and 64% for sadness. These findings establish a Pareto-optimal tradeoff between model size and accuracy, enabling practical affect recognition on resource-constrained mobile devices.

Mobile-Efficient Speech Emotion Recognition Using DistilHuBERT: A Cross-Corpus Validation Study

TL;DR

This work targets mobile-friendly Speech Emotion Recognition by deploying DistilHuBERT, a compact, 8-bit quantized transformer, achieving 23 MB and about 91% of full-scale Wav2Vec 2.0 performance under a strict 5-fold LOSO evaluation on IEMOCAP. It introduces cross-corpus regularization with CREMA-D to improve generalization, yielding a 1.2% gain in Weighted Accuracy and a 32% reduction in cross-fold variance, with notable Neutral class improvement but some Sadness recall decline due to a theatricality gap. Cross-corpus testing on RAVDESS reveals arousal-focused clustering of errors rather than random misclassification, with strong arousal detection (Anger recall 97%, Sadness 64%), illustrating domain shift effects. The paper also proposes a practical deployment architecture leveraging VAD, windowed processing, and temporal pooling to enable robust, privacy-preserving on-device SER on resource-constrained mobile devices.

Abstract

Speech Emotion Recognition (SER) has significant potential for mobile applications, yet deployment remains constrained by the computational demands of state-of-the-art transformer architectures. This paper presents a mobile-efficient SER system based on DistilHuBERT, a distilled and 8-bit quantized transformer that achieves 92% parameter reduction compared to full-scale Wav2Vec 2.0 models while maintaining competitive accuracy. We conduct a rigorous 5-fold Leave-One-Session-Out (LOSO) cross-validation on the IEMOCAP dataset to ensure speaker independence, augmented with cross-corpus training on CREMA-D to enhance generalization. Cross-corpus training with CREMA-D yields a 1.2% improvement in Weighted Accuracy, a 1.4% gain in Macro F1-score, and a 32% reduction in cross-fold variance, with the Neutral class showing the most substantial benefit at 5.4% F1-score improvement. Our approach achieves an Unweighted Accuracy of 61.4% with a quantized model footprint of only 23 MB, representing approximately 91% of full-scale baseline performance. Cross-corpus evaluation on RAVDESS reveals that the theatrical nature of acted emotions causes predictions to cluster by arousal level rather than valence: happiness is systematically confused with anger due to acoustic saturation in high-energy expressions. Despite this theatricality effect reducing overall RAVDESS accuracy to 43.29%, the model maintains robust arousal detection with 97% recall for anger and 64% for sadness. These findings establish a Pareto-optimal tradeoff between model size and accuracy, enabling practical affect recognition on resource-constrained mobile devices.
Paper Structure (17 sections, 1 equation, 1 figure, 3 tables)

This paper contains 17 sections, 1 equation, 1 figure, 3 tables.

Table of Contents

  1. List of Abbreviations
  2. Introduction
  3. Methodology
  4. Experimental Framework and Dataset Integration
  5. Acoustic Processing and Data Augmentation
  6. Feature Extraction and Architecture Selection
  7. Optimization Strategy and Hyperparameters
  8. Results and Discussion
  9. Cross-Corpus Regularization via CREMA-D
  10. Comparative Benchmarking and Methodological Rigor
  11. Architectural Capacity and the Pareto Frontier
  12. Production Model Development and Cross-Corpus Robustness
  13. Production Model Formulation and Performance
  14. Generalization and the Theatricality Effect in Error Analysis
  15. Proposed Deployment Architecture
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Cross-corpus confusion matrix on RAVDESS. The results highlight the model's tendency to cluster high-arousal (Anger/Happy) and low-arousal (Neutral/Sad) states, suggesting that theatrical over-emphasis in the target corpus obscures the subtle valence cues learned during training. Overall Accuracy: 43.29%; Unweighted Recall: 46.18%.