YouTube Comments Decoded: Leveraging LLMs for Low Resource Language Classification
Aniket Deroy, Subhankar Maity
TL;DR
This work tackles sarcasm detection in code-mixed Tamil-English and Malayalam-English YouTube comments by introducing a gold-standard corpus and evaluating a prompt-based approach using GPT-3.5 Turbo in zero-shot mode. The methodology emphasizes language-specific prompts and does not rely on fine-tuning, reporting macro-F1 scores of 0.61 for Tamil and 0.50 for Malayalam, with Tamil ranking higher in a competitive setting. The results reveal robust non-sarcastic detection but substantial challenges in detecting sarcasm, especially in Malayalam, highlighting language-specific difficulties in code-mixed contexts. Overall, the paper demonstrates the viability of prompt-based strategies for under-resourced, code-mixed languages and provides a dataset to guide future improvements in sarcasm and sentiment analysis for multilingual social media content.
Abstract
Sarcasm detection is a significant challenge in sentiment analysis, particularly due to its nature of conveying opinions where the intended meaning deviates from the literal expression. This challenge is heightened in social media contexts where code-mixing, especially in Dravidian languages, is prevalent. Code-mixing involves the blending of multiple languages within a single utterance, often with non-native scripts, complicating the task for systems trained on monolingual data. This shared task introduces a novel gold standard corpus designed for sarcasm and sentiment detection within code-mixed texts, specifically in Tamil-English and Malayalam-English languages. The primary objective of this task is to identify sarcasm and sentiment polarity within a code-mixed dataset of Tamil-English and Malayalam-English comments and posts collected from social media platforms. Each comment or post is annotated at the message level for sentiment polarity, with particular attention to the challenges posed by class imbalance, reflecting real-world scenarios.In this work, we experiment with state-of-the-art large language models like GPT-3.5 Turbo via prompting to classify comments into sarcastic or non-sarcastic categories. We obtained a macro-F1 score of 0.61 for Tamil language. We obtained a macro-F1 score of 0.50 for Malayalam language.