INDIC QA BENCHMARK: A Multilingual Benchmark to Evaluate Question Answering capability of LLMs for Indic Languages

TL;DR

Indic-QA introduces the largest publicly available context-grounded QA benchmark for 11 Indic languages, combining translated English QA datasets, native Indic datasets, and Gemini-generated synthetic data to evaluate extractive and abstractive QA. The authors benchmark a wide range of multilingual LLMs and instruction-tuned variants, comparing zero-shot and few-shot performance and contrasting direct source-language inference with the Translate-Test paradigm. They find that translation-based approaches substantially improve QA for low-resource languages, while high-resource languages benefit from direct inference; instruction-finetuning generally helps abstractive QA, though effects vary by model. The work highlights persistent English bias in multilingual models, emphasizes robust translation pipelines, and provides a valuable resource to spur research into QA capabilities in underrepresented languages.

Abstract

Large Language Models (LLMs) perform well on unseen tasks in English, but their abilities in non English languages are less explored due to limited benchmarks and training data. To bridge this gap, we introduce the Indic QA Benchmark, a large dataset for context grounded question answering in 11 major Indian languages, covering both extractive and abstractive tasks. Evaluations of multilingual LLMs, including instruction finetuned versions, revealed weak performance in low resource languages due to a strong English language bias in their training data. We also investigated the Translate Test paradigm,where inputs are translated to English for processing and the results are translated back into the source language for output. This approach outperformed multilingual LLMs, particularly in low resource settings. By releasing Indic QA, we aim to promote further research into LLMs question answering capabilities in low resource languages. This benchmark offers a critical resource to address existing limitations and foster multilingual understanding.

Figures (17)

• Figure 1: Total tokens available for each Indian language in the Sangraha Data rahman2024indicllmsuite. In contrast, RefinedWeb penedo2023refinedwebdatasetfalconllm contains around 5 Trillion tokens in English.
• Figure 2: Comparison of LLama 3-8B evaluation results using source language test set vs Translate test set. The results clearly indicate that the Translate-Test paradigm yields better scores for low-resource languages (Punjabi, Odia, Assamese), whereas the source language test set gets better scores for mid-resource language Hindi and Marathi which has high correlation with Hindi.
• Figure 3: Workflow of synthetic data creation, in the fig. LLM used is Gemini-pro model.
• Figure 4: Inference in source Language (Top) vs Translate Test Inference (Bottom).
• Figure 5: Evaluation prompt used for the base model.