A Baseline Multimodal Approach to Emotion Recognition in Conversations

TL;DR

This work addresses emotion recognition in conversational data by proposing a lightweight, reproducible multimodal baseline that fuses transformer-based text representations with self-supervised speech features. Using SemEval-2024 Task 3 on the Friends dataset, the study systematically compares text-only and audio-only models and demonstrates the superior performance of a late-fusion ensemble (RoBERTa + Wav2Vec2) with an accuracy of 62.97%. Text models consistently outperform audio models, but the multimodal ensemble yields the best overall results, highlighting the value of integrating semantic and paralinguistic cues. As a transparent reference point with limited tuning, this baseline supports future, more rigorous comparisons and motivates further exploration of additional modalities and cross-domain generalization.

Abstract

We present a lightweight multimodal baseline for emotion recognition in conversations using the SemEval-2024 Task 3 dataset built from the sitcom Friends. The goal of this report is not to propose a novel state-of-the-art method, but to document an accessible reference implementation that combines (i) a transformer-based text classifier and (ii) a self-supervised speech representation model, with a simple late-fusion ensemble. We report the baseline setup and empirical results obtained under a limited training protocol, highlighting when multimodal fusion improves over unimodal models. This preprint is provided for transparency and to support future, more rigorous comparisons.