MERaLiON-SER: Robust Speech Emotion Recognition Model for English and SEA Languages
Hardik B. Sailor, Aw Ai Ti, Chen Fang Yih Nancy, Chiu Ying Lay, Ding Yang, He Yingxu, Jiang Ridong, Li Jingtao, Liao Jingyi, Liu Zhuohan, Lu Yanfeng, Ma Yi, Manas Gupta, Muhammad Huzaifah Bin Md Shahrin, Nabilah Binte Md Johan, Nattadaporn Lertcheva, Pan Chunlei, Pham Minh Duc, Siti Maryam Binte Ahmad Subaidi, Siti Umairah Binte Mohammad Salleh, Sun Shuo, Tarun Kumar Vangani, Wang Qiongqiong, Won Cheng Yi Lewis, Wong Heng Meng Jeremy, Wu Jinyang, Zhang Huayun, Zhang Longyin, Zou Xunlong
TL;DR
MERaLiON-SER addresses robust speech emotion recognition across English and Southeast Asian languages by coupling a frozen Whisper-Medium encoder with a parameter-efficient ECAPA-TDNN-style downstream head and LoRA adapters. It jointly optimizes discrete seven-class emotion classification and three-dimensional affective scores via a composite loss $ ext{L} = ext{λ}_{ ext{cat}} ext{L}_{ ext{CE}} + ext{λ}_{ ext{dim}} ext{L}_{ ext{CCC}}$, achieving strong cross-lingual generalization on Singaporean datasets and public multilingual benchmarks. Across comparisons with open-source speech encoders, open-source Audio-LLMs, and closed-source multimodal models, MERaLiON-SER-v1 consistently yields higher unweighted average recall (UAR) and demonstrates that task-specific, speech-only models remain superior for fine-grained paralinguistic reasoning. The work highlights the practical potential of emotion-aware perception in agentic audio systems and sets a foundation for scalable, empathetic multimodal AI that operates effectively in multilingual, code-mixed environments.
Abstract
We present MERaLiON-SER, a robust speech emotion recognition model designed for English and Southeast Asian languages. The model is trained using a hybrid objective combining weighted categorical cross-entropy and Concordance Correlation Coefficient (CCC) losses for joint discrete and dimensional emotion modelling. This dual approach enables the model to capture both the distinct categories of emotion (like happy or angry) and the fine-grained, such as arousal (intensity), valence (positivity/negativity), and dominance (sense of control), leading to a more comprehensive and robust representation of human affect. Extensive evaluations across multilingual Singaporean languages (English, Chinese, Malay, and Tamil ) and other public benchmarks show that MERaLiON-SER consistently surpasses both open-source speech encoders and large Audio-LLMs. These results underscore the importance of specialised speech-only models for accurate paralinguistic understanding and cross-lingual generalisation. Furthermore, the proposed framework provides a foundation for integrating emotion-aware perception into future agentic audio systems, enabling more empathetic and contextually adaptive multimodal reasoning.