OmniMER: Indonesian Multimodal Emotion Recognition via Auxiliary-Enhanced LLM Adaptation

TL;DR

This work introduces IndoMER, the first Indonesian multimodal emotion recognition benchmark, and OmniMER, a framework that strengthens unimodal emotional grounding via auxiliary perception tasks (text keywords, facial expressions, and prosody) before multimodal fusion. By leveraging an omni-modal LLM backbone with instruction-conditioned prompting and LoRA-based fine-tuning, OmniMER achieves substantial Macro-F1 gains on both sentiment and emotion recognition in IndoMER, particularly improving minority classes. Cross-lingual validation on CH-SIMS demonstrates transferability across languages with varying emotional expressions. The study contributes a publicly available dataset, a novel auxiliary-task adaptation strategy, and evidence of practical impact for low-resource, cross-lingual MER applications.

Abstract

Indonesian, spoken by over 200 million people, remains underserved in multimodal emotion recognition research despite its dominant presence on Southeast Asian social media platforms. We introduce IndoMER, the first multimodal emotion recognition benchmark for Indonesian, comprising 1,944 video segments from 203 speakers with temporally aligned text, audio, and visual annotations across seven emotion categories. The dataset exhibits realistic challenges including cross-modal inconsistency and long-tailed class distributions shaped by Indonesian cultural communication norms. To address these challenges, we propose OmniMER, a multimodal adaptation framework built upon Qwen2.5-Omni that enhances emotion recognition through three auxiliary modality-specific perception tasks: emotion keyword extraction for text, facial expression analysis for video, and prosody analysis for audio. These auxiliary tasks help the model identify emotion-relevant cues in each modality before fusion, reducing reliance on spurious correlations in low-resource settings. Experiments on IndoMER show that OmniMER achieves 0.582 Macro-F1 on sentiment classification and 0.454 on emotion recognition, outperforming the base model by 7.6 and 22.1 absolute points respectively. Cross-lingual evaluation on the Chinese CH-SIMS dataset further demonstrates the generalizability of the proposed framework. The dataset and code are publicly available. https://github.com/yanxm01/INDOMER

Figures (5)

• Figure 1: Categorical breakdown of video themes within the IndoMER dataset. The dataset spans thirteen distinct topics, ranging from personal life sharing to specific domains like health and politics. This wide variety of themes is selected to capture rich emotional nuances in different contexts.
• Figure 2: Distribution of emotional and sentiment annotations in the IndoMER dataset. (a) Sentiment label distributions across text, audio, and visual modalities, compared with the multimodal consensus (ground truth). (b) Detailed breakdown of emotion categories within the multimodal annotations, highlighting the long-tailed nature of natural emotional communication.
• Figure 3: Overview of the proposed OmniMER framework. It leverages Qwen2.5-Omni as the unified multimodal backbone and incorporates modality-specific auxiliary tasks (text keywords, facial expressions, and speaker intonation) to enhance unimodal emotion grounding before multimodal fusion.
• Figure 4: Construction of modality-specific auxiliary supervision from sentiment labels. Explanations are retained only when their associated sentiment predictions are consistent with ground-truth annotations, or regenerated under sentiment constraints otherwise.
• Figure 5: A representative example illustrating how modality-specific auxiliary tasks expose complementary and conflicting emotional cues across text, video, and audio, and how OmniMER integrates these cues to produce a robust multimodal emotion prediction.