IndiBias: A Benchmark Dataset to Measure Social Biases in Language Models for Indian Context
Nihar Ranjan Sahoo, Pranamya Prashant Kulkarni, Narjis Asad, Arif Ahmad, Tanu Goyal, Aparna Garimella, Pushpak Bhattacharyya
TL;DR
IndiBias tackles the lack of India-context bias benchmarks by building a bilingual English–Hindi dataset that combines Indian CrowS-Pairs translations, a bias-tuple generator, and LLM-aided sentence construction. It advances a robust pipeline for cross-lingual bias evaluation and analyzes ten multilingual LLMs, revealing persistent biases across gender, religion, caste, and their intersections. The dataset comprises 800 ICS sentence pairs and 300 bias tuples, augmented by bleached templates and SEAT-based intersectional analyses, enabling both direct bias measurement and cross-language comparisons. This work provides a practical tool for benchmarking and debiasing Indian-language LLMs, with implications for more culturally aware NLP systems and policy-relevant fairness assessments.
Abstract
The pervasive influence of social biases in language data has sparked the need for benchmark datasets that capture and evaluate these biases in Large Language Models (LLMs). Existing efforts predominantly focus on English language and the Western context, leaving a void for a reliable dataset that encapsulates India's unique socio-cultural nuances. To bridge this gap, we introduce IndiBias, a comprehensive benchmarking dataset designed specifically for evaluating social biases in the Indian context. We filter and translate the existing CrowS-Pairs dataset to create a benchmark dataset suited to the Indian context in Hindi language. Additionally, we leverage LLMs including ChatGPT and InstructGPT to augment our dataset with diverse societal biases and stereotypes prevalent in India. The included bias dimensions encompass gender, religion, caste, age, region, physical appearance, and occupation. We also build a resource to address intersectional biases along three intersectional dimensions. Our dataset contains 800 sentence pairs and 300 tuples for bias measurement across different demographics. The dataset is available in English and Hindi, providing a size comparable to existing benchmark datasets. Furthermore, using IndiBias we compare ten different language models on multiple bias measurement metrics. We observed that the language models exhibit more bias across a majority of the intersectional groups.