MILU: A Multi-task Indic Language Understanding Benchmark

TL;DR

MILU tackles the gap in evaluating Large Language Models on Indic languages by introducing an India-centric benchmark spanning 11 languages, 8 domains, and 41 subjects drawn from real exams. It assembles roughly 79K MCQ items across 8 domains and 41 subjects to test both linguistic competence and culturally specific knowledge, evaluating 42 models with 0/1/5-shot settings and log-likelihood scoring. The study finds GPT-4o leading at about $74.7\%$, but many models lag, with open multilingual models outperforming language-specific variants and significant domain- and resource-based gaps, especially in Arts, Humanities, and Law. By releasing all data, code, and artifacts, MILU provides a foundational benchmark to spur development of more inclusive, culturally aware Indic-language LLMs.

Abstract

Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU, a Multi task Indic Language Understanding Benchmark, a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting both general and culturally specific knowledge. With an India-centric design, incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science and mathematics. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 74 percent. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high resource languages as compared to low resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts and Humanities, Law and Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts are publicly available to foster open research.

Figures (6)

• Figure 1: Average performance of all the evaluated models on MILU. The closed models are shown in Orange, the open models are shown in Green, and the language-specific models are shown in Blue.
• Figure 2: Distribution of the number of questions across different domains, averaged across all languages. Refer to Section (§\ref{['sec: analysis']}) for more details.
• Figure 3: Comparison of Base and Instruct models averaged across all languages for varying number of in-context examples. We plot the average accuracies of the Gemma and Llama series of models, highlighting the performance trend as the number of in-context examples increases. Refer to Section (§\ref{['sec: shots']}) for more details.
• Figure 4: Evaluation results of Base models (\ref{['fig:img1']}) and Instruct models (\ref{['fig:img2']}) on the different domains supported in MILU. The plot shows the average 5-shot accuracies across all languages for various models. Refer to Section (§\ref{['sec: domains']})
• Figure 5: Comparison of performance of Gemma and Llama family of models across different parameter scales. We plot the zero-shot average accuracies of all models across languages. Refer to Section (§\ref{['sec: scale']}) for more details.