Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wav2Small: Distilling Wav2Vec2 to 72K parameters for Low-Resource Speech emotion recognition

Dionyssos Kounadis-Bastian, Oliver Schrüfer, Anna Derington, Hagen Wierstorf, Florian Eyben, Felix Burkhardt, Björn Schuller

TL;DR

A large Transformer Sota A/D/V model as Teacher/Annotator is used to train 5 student models: 4 MobileNets and the proposed Wav2Small, using only the Teacher's A/D/V outputs instead of human annotations, a potential solution for A/D/V on hardware with low resources.

Abstract

Speech Emotion Recognition (SER) needs high computational resources to overcome the challenge of substantial annotator disagreement. Today SER is shifting towards dimensional annotations of arousal, dominance, and valence (A/D/V). Universal metrics as the L2 distance prove unsuitable for evaluating A/D/V accuracy due to non converging consensus of annotator opinions. However, Concordance Correlation Coefficient (CCC) arose as an alternative metric for A/D/V where a model's output is evaluated to match a whole dataset's CCC rather than L2 distances of individual audios. Recent studies have shown that wav2vec2 / wavLM architectures outputing a float value for each A/D/V dimension achieve today's State-of-the-art (Sota) CCC on A/D/V. The Wav2Vec2.0 / WavLM family has a high computational footprint, but training small models using human annotations has been unsuccessful. In this paper we use a large Transformer Sota A/D/V model as Teacher/Annotator to train 5 student models: 4 MobileNets and our proposed Wav2Small, using only the Teacher's A/D/V outputs instead of human annotations. The Teacher model we propose also sets a new Sota on the MSP Podcast dataset of valence CCC=0.676. We choose MobileNetV4 / MobileNet-V3 as students, as MobileNet has been designed for fast execution times. We also propose Wav2Small - an architecture designed for minimal parameters and RAM consumption. Wav2Small with an .onnx (quantised) of only 120KB is a potential solution for A/D/V on hardware with low resources, having only 72K parameters vs 3.12M parameters for MobileNet-V4-Small.

Wav2Small: Distilling Wav2Vec2 to 72K parameters for Low-Resource Speech emotion recognition

TL;DR

A large Transformer Sota A/D/V model as Teacher/Annotator is used to train 5 student models: 4 MobileNets and the proposed Wav2Small, using only the Teacher's A/D/V outputs instead of human annotations, a potential solution for A/D/V on hardware with low resources.

Abstract

Speech Emotion Recognition (SER) needs high computational resources to overcome the challenge of substantial annotator disagreement. Today SER is shifting towards dimensional annotations of arousal, dominance, and valence (A/D/V). Universal metrics as the L2 distance prove unsuitable for evaluating A/D/V accuracy due to non converging consensus of annotator opinions. However, Concordance Correlation Coefficient (CCC) arose as an alternative metric for A/D/V where a model's output is evaluated to match a whole dataset's CCC rather than L2 distances of individual audios. Recent studies have shown that wav2vec2 / wavLM architectures outputing a float value for each A/D/V dimension achieve today's State-of-the-art (Sota) CCC on A/D/V. The Wav2Vec2.0 / WavLM family has a high computational footprint, but training small models using human annotations has been unsuccessful. In this paper we use a large Transformer Sota A/D/V model as Teacher/Annotator to train 5 student models: 4 MobileNets and our proposed Wav2Small, using only the Teacher's A/D/V outputs instead of human annotations. The Teacher model we propose also sets a new Sota on the MSP Podcast dataset of valence CCC=0.676. We choose MobileNetV4 / MobileNet-V3 as students, as MobileNet has been designed for fast execution times. We also propose Wav2Small - an architecture designed for minimal parameters and RAM consumption. Wav2Small with an .onnx (quantised) of only 120KB is a potential solution for A/D/V on hardware with low resources, having only 72K parameters vs 3.12M parameters for MobileNet-V4-Small.
Paper Structure (25 sections, 5 figures, 1 table)

This paper contains 25 sections, 5 figures, 1 table.

Table of Contents

  1. INTRODUCTION
  2. Wav2Small
  3. A / D / V Distillation
  4. Teacher
  5. Data / Augmentation
  6. Quadrant Correction Loss Function
  7. Experiments
  8. Ablations
  9. Vectorisation of Time as Channels
  10. A/D/V clusters of categorical emotions
  11. Outlook
  12. CONCLUSION
  13. Why are high computational resources needed to overcome annotator disagreement, what performance do we get if we train the smaller models from scratch without distillation?
  14. '..VGG has a low RAM footprint from absence of skip connections..' Why do skip connections use a lot of RAM? Also, 36 MB of RAM used by MobileNetV4-S seems already small?
  15. 'The VGG7 of Wav2Vec2 has the same execution time as 6 transformer layers?
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Arousal-Valence predictions for the Crema-d cao2014crema test set, colouring according to Crema-d ground truth categories.
  • Figure 2: CCC achieved by all tested models on MSP Podcast v1.7 Test1 - original human annotations.
  • Figure 3: Wav2Small discrepancy from teacher for the Japanese audio track of Harry Potter vol1. Not included for train.
  • Figure 4: CCC achieved by all tested models on IEMOCAP - original human annotations. Audio of all 5 Sessions has been concatenated to a single dataset.
  • Figure 5: Execution time of the 2 parts of Wav2Vec2 wagner23dhsu2021robust: The Feature Extractor (VGG7) and The Transformer-Encoder (prunned to 6 Transformer layers). Surprisingly, the VGG7 is slower than 6 Transformer layers for audio input $\ge 2s$.