Sociolinguistically Informed Interpretability: A Case Study on Hinglish Emotion Classification

TL;DR

This work investigates how sociolinguistic patterns in Hinglish influence emotion classification by fine-tuning three pre-trained language models on a Hinglish emotion dataset and examining token-level contributions with LIME alongside token language IDs. The study finds that English tokens bias predictions toward positive emotions while Hindi tokens push toward negative emotions, with Hindi swear words exerting particularly strong negative influence. Importantly, pre-training on code-mixed data augments the learning of these associations when task data is scarce, though models can overgeneralize these heuristics to infrequent cases. The findings bridge sociolinguistics and interpretability, offering guidance for developing robust code-mixed NLP systems and cautioning against unchecked generalizations of language-based cues.

Abstract

Emotion classification is a challenging task in NLP due to the inherent idiosyncratic and subjective nature of linguistic expression, especially with code-mixed data. Pre-trained language models (PLMs) have achieved high performance for many tasks and languages, but it remains to be seen whether these models learn and are robust to the differences in emotional expression across languages. Sociolinguistic studies have shown that Hinglish speakers switch to Hindi when expressing negative emotions and to English when expressing positive emotions. To understand if language models can learn these associations, we study the effect of language on emotion prediction across 3 PLMs on a Hinglish emotion classification dataset. Using LIME and token level language ID, we find that models do learn these associations between language choice and emotional expression. Moreover, having code-mixed data present in the pre-training can augment that learning when task-specific data is scarce. We also conclude from the misclassifications that the models may overgeneralise this heuristic to other infrequent examples where this sociolinguistic phenomenon does not apply.

Figures (5)

• Figure 1: Our workflow. We train 3 emotion classification models, then obtain LIME scores for each token (positive scores in red, negative scores in blue, and zero scores in grey). These same samples are then tagged with token-level language ID, which enables us to examine LIME distributional differences by language.
• Figure 2: Frequencies of Hindi (green), English (purple) and Other (orange) tokens to be assigned a positive (solid) or a negative (striped) LIME score for examples predicted as joy and anger, for all models.
• Figure 3: An example from the dataset labelled as joy, with the translation and language ID tags. The 3 tokens with the highest LIME scores are marked in blue, and the 3 tokens with the lowest scores are marked in red.
• Figure 4: Confusion matrix containing the percentage of correctly and incorrectly classified examples for each label combination. The blue cells represent correct classifications, and the pink cells represent incorrect classifications.
• Figure 5: An example labelled anger that was misclassified as joy owing to the English phrase (English - purple; Hindi - green; Other - orange) in the sentence having a positive connotation, even though the sentence itself conveys anger.