BnMMLU: Measuring Massive Multitask Language Understanding in Bengali

TL;DR

BnMMLU addresses the underrepresentation of Bengali in large-scale multilingual evaluation by introducing a 41-domain, $134{,}375$-item MCQ benchmark that preserves MathML content. The authors provide a rigorous data pipeline (OCR/post-correction, de-duplication, and contamination checks) and a HARD subset to stress-test models, evaluating $24$ models across $11$ families under Direct/CoT and 0-shot/5-shot with Reasoning-On vs Non-Reasoning. Key findings show proprietary models lead overall, open-weight models narrow the gap, and reasoning-enabled prompts yield substantial gains, especially on HARD items, with diminishing returns as scale grows. The work contributes a reproducible Bengali knowledge-evaluation framework and highlights domain-specific failure modes and prompt-design considerations to advance multilingual NLP for Bengali.

Abstract

Large-scale multitask benchmarks have driven rapid progress in language modeling, yet most emphasize high-resource languages such as English, leaving Bengali underrepresented. We present BnMMLU, a comprehensive benchmark for measuring massive multitask language understanding in Bengali. BnMMLU spans 41 domains across STEM, humanities, social sciences, and general knowledge, and contains 134,375 multiple-choice question-option pairs--the most extensive Bengali evaluation suite to date. The dataset preserves mathematical content via MathML, and includes BnMMLU-HARD, a compact subset constructed from questions most frequently missed by top systems to stress difficult cases. We benchmark 24 model variants across 11 LLM families, spanning open-weights general/multilingual, Bengali-centric open-weights, and proprietary models, covering multiple parameter scales and instruction-tuned settings. We evaluate models under standardized protocols covering two prompting styles (Direct vs. Chain-of-Thought) and two context regimes (0-shot vs. 5-shot), reporting accuracy consistently across families. Our analysis highlights persistent gaps in reasoning and application skills and indicates sublinear returns to scale across model sizes. We release the dataset and evaluation templates to support rigorous, reproducible assessment of Bengali language understanding and to catalyze progress in multilingual NLP.

Figures (12)

• Figure 1: An overview of the pipeline for constructing the BnMMLU benchmark.
• Figure 2: Error rate trends by question length (in characters) across ten evaluated models on the BnMMLU-FULL benchmark. Each subplot represents an individual model, with the x-axis indicating question length bins and the y-axis showing corresponding error rates. Overall accuracy for each model is annotated in its respective panel for reference.
• Figure 3: Subdomain difficulty versus cross-model consistency on the BnMMLU-FULL benchmark under 0-shot Direct prompting. The x-axis shows mean accuracy across models (higher = easier), and the y-axis shows standard deviation (higher = more inconsistent); each point is a subdomain color-coded by difficulty bucket (Easy, Medium, Hard). The four quadrants (Easy & Consistent, Easy but Inconsistent, Difficult and Inconsistent, Difficult but Consistent) summarize how subdomain complexity and variability interact in assessing LLM robustness.
• Figure 4: Prompt used for Bengali copy-editing, formatted consistently with our evaluation prompt boxes.
• Figure 5: Sample scanned pages of Bengali multiple-choice questions collected from academic and preparatory guidebooks.