cantnlp@DravidianLangTech2025: A Bag-of-Sounds Approach to Multimodal Hate Speech Detection

TL;DR

This study explores a bag-of-sounds approach to multimodal hate speech detection in Dravidian languages by converting speech to Mel spectrogram features and training audio-based classifiers within the MSMDA-DL task. Despite promising development results for Malayalam and Tamil, the model performance on the test set was below chance, revealing substantial data and generalization challenges for multimodal setups. The authors compare several statistical classifiers, emphasize the importance of macro F1 as the evaluation metric, and highlight data imbalance and potential overfitting as key issues. The work demonstrates that audio signals carry valuable paralinguistic information for hate speech detection and lays groundwork for future integration of richer features and models, including transformer-based or sociolinguistically informed approaches, once better-balanced data become available.

Abstract

This paper presents the systems and results for the Multimodal Social Media Data Analysis in Dravidian Languages (MSMDA-DL) shared task at the Fifth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages (DravidianLangTech-2025). We took a `bag-of-sounds' approach by training our hate speech detection system on the speech (audio) data using transformed Mel spectrogram measures. While our candidate model performed poorly on the test set, our approach offered promising results during training and development for Malayalam and Tamil. With sufficient and well-balanced training data, our results show that it is feasible to use both text and speech (audio) data in the development of multimodal hate speech detection systems.