IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understanding

TL;DR

IndicMMLU-Pro extends the MMLU-Pro multitask benchmark to nine Indic languages by translating the English dataset with IndicTrans2 and enforcing rigorous back-translation quality checks and expert proofreading. It benchmarks a broad set of multilingual and Indic-focused models to reveal strong GPT-4o performance and pronounced language-specific variability driven by script and language-family differences, while highlighting gaps in complete cross-language evaluation. The framework demonstrates the importance of high-quality translation pipelines, multi-metric evaluation, and dataset availability for robust Indic-language AI benchmarking, guiding future data collection, cross-lingual transfer, and fine-tuning strategies. Altogether, IndicMMLU-Pro provides a standardized, culturally aware platform to drive progress in Indic language understanding, reasoning, and generation at scale.

Abstract

Known by more than 1.5 billion people in the Indian subcontinent, Indic languages present unique challenges and opportunities for natural language processing (NLP) research due to their rich cultural heritage, linguistic diversity, and complex structures. IndicMMLU-Pro is a comprehensive benchmark designed to evaluate Large Language Models (LLMs) across Indic languages, building upon the MMLU Pro (Massive Multitask Language Understanding) framework. Covering major languages such as Hindi, Bengali, Gujarati, Marathi, Kannada, Punjabi, Tamil, Telugu, and Urdu, our benchmark addresses the unique challenges and opportunities presented by the linguistic diversity of the Indian subcontinent. This benchmark encompasses a wide range of tasks in language comprehension, reasoning, and generation, meticulously crafted to capture the intricacies of Indian languages. IndicMMLU-Pro provides a standardized evaluation framework to push the research boundaries in Indic language AI, facilitating the development of more accurate, efficient, and culturally sensitive models. This paper outlines the benchmarks' design principles, task taxonomy, and data collection methodology, and presents baseline results from state-of-the-art multilingual models.

Figures (9)

• Figure 1: IndicMMLU-Pro Dataset Construction and Evaluation Pipeline. The diagram illustrates the end-to-end process of creating and validating the IndicMMLU-Pro dataset across nine Indic languages. Starting with the English MMLU-Pro dataset, content is translated using IndicTrans2 (1B parameters) and undergoes rigorous quality assurance through back-translation and multiple metric evaluations (chrF++, BLEU, METEOR, TER, and SacreBLEU). Only translations meeting quality thresholds proceed to the final dataset. The workflow also shows the comprehensive evaluation process including expert proofreading involving 13 reviewers who assess semantic accuracy, fluency, and linguistic style. This systematic approach ensures the creation of a high-quality, multilingual benchmark dataset that maintains the integrity of the original MMLU-Pro while adapting to the linguistic nuances of Indic languages.
• Figure 2: The original text sample, its Hindi translation, and the corresponding back-translated text
• Figure 3: The original text sample, its Gujarati translation, and the corresponding back-translated text
• Figure 4: The original text sample, its Tamil translation, and the corresponding back-translated text
• Figure 5: Model Accuracy Across Different Languages