Qieemo: Speech Is All You Need in the Emotion Recognition in Conversations
Jinming Chen, Jingyi Fang, Yuanzhong Zheng, Yaoxuan Wang, Haojun Fei
TL;DR
This work tackles emotion recognition in conversations by avoiding reliance on multiple modalities. It introduces Qieemo, a framework that repurposes a pretrained ASR backbone to extract frame-aligned textual and emotional cues, fusing them with a Multimodal Fusion Module and Cross-modal Attention to achieve robust emotion classification from audio alone. The authors demonstrate, on the IEMOCAP dataset, that their unimodal (audio-only) approach surpasses state-of-the-art unimodal, multimodal, and SSL baselines, validating the effectiveness of cross-modal fusion built atop ASR features. The contributions include identifying the most emotion-discriminative ASR blocks, validating MMF/CMA fusion, and proving the approach’s universality across ASR backbones, with practical implications for real-time emotion recognition in conversational systems.
Abstract
Emotion recognition plays a pivotal role in intelligent human-machine interaction systems. Multimodal approaches benefit from the fusion of diverse modalities, thereby improving the recognition accuracy. However, the lack of high-quality multimodal data and the challenge of achieving optimal alignment between different modalities significantly limit the potential for improvement in multimodal approaches. In this paper, the proposed Qieemo framework effectively utilizes the pretrained automatic speech recognition (ASR) model backbone which contains naturally frame aligned textual and emotional features, to achieve precise emotion classification solely based on the audio modality. Furthermore, we design the multimodal fusion (MMF) module and cross-modal attention (CMA) module in order to fuse the phonetic posteriorgram (PPG) and emotional features extracted by the ASR encoder for improving recognition accuracy. The experimental results on the IEMOCAP dataset demonstrate that Qieemo outperforms the benchmark unimodal, multimodal, and self-supervised models with absolute improvements of 3.0%, 1.2%, and 1.9% respectively.