Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture

Arijit Maji, Raghvendra Kumar, Akash Ghosh, Anushka, Sriparna Saha

TL;DR

SANSKRITI introduces a large-scale benchmark to evaluate language models’ knowledge of Indian culture, addressing a critical gap in culturally nuanced AI. The dataset comprises 21,853 manually curated MCQs spanning 28 states and 8 union territories across 16 cultural attributes, with zero-shot evaluation across LLMs, ILMs, and SLMs. Key findings show substantial performance disparities, with GPT-4o leading overall and regional- and attribute-specific gaps revealing limitations in region-specific knowledge and training data. The work provides a public resource and a framework for analyzing cultural competence, and it outlines plans to broaden attributes, add multilingual and multimodal extensions, and enable more inclusive AI in culturally diverse contexts.

Abstract

Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.

SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culture

TL;DR

SANSKRITI introduces a large-scale benchmark to evaluate language models’ knowledge of Indian culture, addressing a critical gap in culturally nuanced AI. The dataset comprises 21,853 manually curated MCQs spanning 28 states and 8 union territories across 16 cultural attributes, with zero-shot evaluation across LLMs, ILMs, and SLMs. Key findings show substantial performance disparities, with GPT-4o leading overall and regional- and attribute-specific gaps revealing limitations in region-specific knowledge and training data. The work provides a public resource and a framework for analyzing cultural competence, and it outlines plans to broaden attributes, add multilingual and multimodal extensions, and enable more inclusive AI in culturally diverse contexts.

Abstract

Language Models (LMs) are indispensable tools shaping modern workflows, but their global effectiveness depends on understanding local socio-cultural contexts. To address this, we introduce SANSKRITI, a benchmark designed to evaluate language models' comprehension of India's rich cultural diversity. Comprising 21,853 meticulously curated question-answer pairs spanning 28 states and 8 union territories, SANSKRITI is the largest dataset for testing Indian cultural knowledge. It covers sixteen key attributes of Indian culture: rituals and ceremonies, history, tourism, cuisine, dance and music, costume, language, art, festivals, religion, medicine, transport, sports, nightlife, and personalities, providing a comprehensive representation of India's cultural tapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), Indic Language Models (ILMs), and Small Language Models (SLMs), revealing significant disparities in their ability to handle culturally nuanced queries, with many models struggling in region-specific contexts. By offering an extensive, culturally rich, and diverse dataset, SANSKRITI sets a new standard for assessing and improving the cultural understanding of LMs.

Paper Structure

This paper contains 24 sections, 15 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Culture in LLMs
  4. NLP for Indian Languages
  5. Collaborative Dataset Creation
  6. Development of SANSKRITI
  7. Data Collection
  8. Annotation Process
  9. Statistical details of SANSKRITI
  10. Experimental Section
  11. Models
  12. Evaluation Setup
  13. Discussion on Results
  14. Main Results
  15. Performance across Cultural Attributes
  16. ...and 9 more sections

Figures (15)

  • Figure 1: Illustration showcasing crafting questions on Chhattisgarh's dance and music Heritage.
  • Figure 2: Distribution of Question Types Showing Balanced Representation Across Categories
  • Figure 3: A map of India illustrating the distribution of questions across all states and union territories.
  • Figure 4: Distribution of Question Types across various Attributes.
  • Figure 5: Word Cloud Representation of Question Categories
  • ...and 10 more figures