Enhancing Speech Emotion Recognition Through Differentiable Architecture Search

TL;DR

The paper addresses the challenge of finding effective neural architectures for Speech Emotion Recognition by applying Differentiable Architecture Search (DARTS) to optimize a joint CNN–LSTM model. By removing fixed CNN layer ordering and allowing DARTS to select CNN operations and their arrangement, the method achieves superior SER accuracy on IEMOCAP and MSP-IMPROV compared with hand-engineered CNN–LSTM baselines and prior NAS approaches, using MFCC features and attention-enhanced LSTM. Key contributions include a novel, autonomous CNN operation selection within a DARTS cell, a four-cell CNN–LSTM search that balances expressiveness and generalization, and empirical evidence that architecture search can yield compact, high-performing SER models. The findings advance SER by demonstrating that differentiable NAS can automate architectural refinement, reducing manual design effort while improving performance.

Abstract

Speech Emotion Recognition (SER) is a critical enabler of emotion-aware communication in human-computer interactions. Recent advancements in Deep Learning (DL) have substantially enhanced the performance of SER models through increased model complexity. However, designing optimal DL architectures requires prior experience and experimental evaluations. Encouragingly, Neural Architecture Search (NAS) offers a promising avenue to determine an optimal DL model automatically. In particular, Differentiable Architecture Search (DARTS) is an efficient method of using NAS to search for optimised models. This paper proposes a DARTS-optimised joint CNN and LSTM architecture, to improve SER performance, where the literature informs the selection of CNN and LSTM coupling to offer improved performance. While DARTS has previously been applied to CNN and LSTM combinations, our approach introduces a novel mechanism, particularly in selecting CNN operations using DARTS. In contrast to previous studies, we refrain from imposing constraints on the order of the layers for the CNN within the DARTS cell; instead, we allow DARTS to determine the optimal layer order autonomously. Experimenting with the IEMOCAP and MSP-IMPROV datasets, we demonstrate that our proposed methodology achieves significantly higher SER accuracy than hand-engineering the CNN-LSTM configuration. It also outperforms the best-reported SER results achieved using DARTS on CNN-LSTM.

Figures (4)

• Figure 1: The proposed model architecture comprises input features processed through CNN, LSTM, and Dense layers, and utilises DARTS for optimising the CNN component.
• Figure 2: DARTS employs steps (a) to (d) to search cell architectures: (a) initialises the graph, (b) forms a search space, (c) updates edge weights, and (d) determines the final cell structure. Edges represent operations, nodes signify representations, with light-coloured edges indicating weaker and dark-coloured edges representing stronger operations.
• Figure 3: Comparison of UA% and Number of Parameters between the DARTS generated (DARTS)($C=4$) and Hand Engineered (HE), CNN, CNN+LSTM and CNN+LSTM with attention models for (a) IEMOCAP and (b) MSP-IMPROV datasets.
• Figure 4: DARTS searched $t^{th}$ cell structure for Normal Cell (Top) Reduction Cell (Bottom) for the CNN+LSTM att. -- DARTS model.