IndRegBias: A Dataset for Studying Indian Regional Biases in English and Code-Mixed Social Media Comments

TL;DR

IndRegBias addresses the under-explored problem of regional biases in Indian English and code-mixed text by introducing a 25,000-comment dataset collected from Reddit and YouTube. It leverages a multilevel annotation policy to label regional bias presence, severity, and targeted region, enabling robust binary and multi-class evaluations. The authors systematically compare zero-shot, few-shot, and fine-tuned LLM/Indic-language models, finding that domain-specific fine-tuning substantially improves bias detection and severity ranking. This dataset and the accompanying evaluation framework provide a valuable resource for developing region-aware NLP models and for studying how current models handle culturally nuanced, multilingual content. The work highlights important practical considerations, including data representativeness and model safety behaviors, that influence bias assessment in diverse linguistic ecosystems.

Abstract

Warning: This paper consists of examples representing regional biases in Indian regions that might be offensive towards a particular region. While social biases corresponding to gender, race, socio-economic conditions, etc., have been extensively studied in the major applications of Natural Language Processing (NLP), biases corresponding to regions have garnered less attention. This is mainly because of (i) difficulty in the extraction of regional bias datasets, (ii) disagreements in annotation due to inherent human biases, and (iii) regional biases being studied in combination with other types of social biases and often being under-represented. This paper focuses on creating a dataset IndRegBias, consisting of regional biases in an Indian context reflected in users' comments on popular social media platforms, namely Reddit and YouTube. We carefully selected 25,000 comments appearing on various threads in Reddit and videos on YouTube discussing trending topics on regional issues in India. Furthermore, we propose a multilevel annotation strategy to annotate the comments describing the severity of regional biased statements. To detect the presence of regional bias and its severity in IndRegBias, we evaluate open-source Large Language Models (LLMs) and Indic Language Models (ILMs) using zero-shot, few-shot, and fine-tuning strategies. We observe that zero-shot and few-shot approaches show lower accuracy in detecting regional biases and severity in the majority of the LLMs and ILMs. However, the fine-tuning approach significantly enhances the performance of the LLM in detecting Indian regional bias along with its severity.

Figures (5)

• Figure 1: Construction of IndRegBias and detecting regional bias with severity using Large Language Models and Indic Language Models. (In the annotation phase, if a comment is regional bias, then the second and third labels are determined. Otherwise, the comment gets a single label of 0. For example, after the data annotation phase, a text/comment is assigned labels as (1, 2, L) where 1 is for stating the comment is regional bias, 2 states the bias is Moderate, and L states the region being targeted.
• Figure 2: Distribution of data across different regions in India.
• Figure 3: Distribution of regional biased comments across Indian states.
• Figure 4: Distribution of severe bias comments across different states.
• Figure 5: Heatmap illustrating regional bias detection accuracy across different models and states.