Emotion Neural Transducer for Fine-Grained Speech Emotion Recognition

TL;DR

The paper tackles the limitation of traditional SER methods that rely on utterance-level labels by introducing Emotion Neural Transducer (ENT), which jointly models speech emotion and ASR to capture fine-grained temporal emotion dynamics. ENT builds an emotion joint network over the neural transducer's alignment lattice to produce a frame-level emotion distribution and employs lattice max pooling to supervise emotional vs non-emotional frames; its extension, Factorized ENT (FENT), separates blank and vocabulary predictions so the blank token also carries emotion information. Empirical results show ENT achieves state-of-the-art performance on IEMOCAP for utterance-level SER and demonstrates strong capability for fine-grained emotion recognition on the ZED dataset, with FENT preserving ASR performance while improving emotion signaling. The approach advances practical SER by enabling transcripts with aligned emotion, improving emotion diarization and enabling more nuanced human–machine interaction.

Abstract

The mainstream paradigm of speech emotion recognition (SER) is identifying the single emotion label of the entire utterance. This line of works neglect the emotion dynamics at fine temporal granularity and mostly fail to leverage linguistic information of speech signal explicitly. In this paper, we propose Emotion Neural Transducer for fine-grained speech emotion recognition with automatic speech recognition (ASR) joint training. We first extend typical neural transducer with emotion joint network to construct emotion lattice for fine-grained SER. Then we propose lattice max pooling on the alignment lattice to facilitate distinguishing emotional and non-emotional frames. To adapt fine-grained SER to transducer inference manner, we further make blank, the special symbol of ASR, serve as underlying emotion indicator as well, yielding Factorized Emotion Neural Transducer. For typical utterance-level SER, our ENT models outperform state-of-the-art methods on IEMOCAP in low word error rate. Experiments on IEMOCAP and the latest speech emotion diarization dataset ZED also demonstrate the superiority of fine-grained emotion modeling. Our code is available at https://github.com/ECNU-Cross-Innovation-Lab/ENT.

Figures (4)

• Figure 1: Emotion Neural Transducer.
• Figure 2: Factorized Emotion Neural Transducer.
• Figure 3: Temporal and Token Lattice Max Pooling Loss.
• Figure 4: Mixing on Emotion lattice.