Common to Whom? Regional Cultural Commonsense and LLM Bias in India

TL;DR

This work investigates sub-national cultural variation in India by introducing Indica, a benchmark designed to test LLMs on region-specific cultural commonsense. By grounding question design in the Outline of Cultural Materials and collecting regionally annotated responses, the authors show that only $39.4\%$ of questions achieve universal agreement across five Indian regions, indicating strong regional heterogeneity. Evaluations across eight LLMs reveal that models generally capture broad cultural concepts but fail to generate region-specific knowledge, and they exhibit geographic bias toward Central and North Indian practices when regional context is absent. Beyond India, the paper provides a generalizable blueprint for evaluating cultural commonsense in other culturally diverse nations, spanning domain selection, data collection, gold-standard establishment, and robust bias measurement. Indica thus argues for culturally competent AI that models within-country diversity rather than assuming national uniformity.

Abstract

Existing cultural commonsense benchmarks treat nations as monolithic, assuming uniform practices within national boundaries. But does cultural commonsense hold uniformly within a nation, or does it vary at the sub-national level? We introduce Indica, the first benchmark designed to test LLMs' ability to address this question, focusing on India - a nation of 28 states, 8 union territories, and 22 official languages. We collect human-annotated answers from five Indian regions (North, South, East, West, and Central) across 515 questions spanning 8 domains of everyday life, yielding 1,630 region-specific question-answer pairs. Strikingly, only 39.4% of questions elicit agreement across all five regions, demonstrating that cultural commonsense in India is predominantly regional, not national. We evaluate eight state-of-the-art LLMs and find two critical gaps: models achieve only 13.4%-20.9% accuracy on region-specific questions, and they exhibit geographic bias, over-selecting Central and North India as the "default" (selected 30-40% more often than expected) while under-representing East and West. Beyond India, our methodology provides a generalizable framework for evaluating cultural commonsense in any culturally heterogeneous nation, from question design grounded in anthropological taxonomy, to regional data collection, to bias measurement.

Figures (9)

• Figure 1: Regional answers to a cultural question and model bias. Each region gives a different answer; models default to Central and North India.
• Figure 2: The Indica creation pipeline: from domain selection to gold standard establishment
• Figure 3: Distribution of 515 questions across 8 domains
• Figure 4: Pairwise and universal agreement rates between all 5 regions. Percentages calculated over questions where both regions provided responses.
• Figure 5: Fully correct accuracy by region on RASA