UrduBench: An Urdu Reasoning Benchmark using Contextually Ensembled Translations with Human-in-the-Loop

TL;DR

UrduBench introduces a standardized Urdu reasoning benchmark by translating MGSM, MATH-500, CommonSenseQA, and OpenBookQA into Urdu using a contextually ensembled translation pipeline with human-in-the-loop validation. It then evaluates a broad spectrum of open-source and instruction-tuned LLMs under multiple prompting strategies, revealing that multi-step and symbolic reasoning in Urdu remains challenging and that language consistency strongly affects performance. The study highlights that explicit chain-of-thought prompting and robust multilingual grounding improve reasoning, while model architecture and scaling alone are insufficient. The framework provides a scalable methodology for reasoning evaluation in Urdu and can be extended to other low-resource languages.

Abstract

Recent advances in large language models (LLMs) have led to strong reasoning capabilities; however, evaluating such models in low-resource languages remains challenging due to the lack of standardized benchmarks. In particular, Urdu reasoning evaluation has been limited by the sensitivity of machine translation and an emphasis on general language tasks rather than reasoning benchmarks. In this paper, we propose a contextually ensembled translation framework with human-in-the-loop validation that leverages multiple translation systems to develop Urdu reasoning benchmarks while preserving contextual and structural integrity. Using this framework, we translate widely adopted reasoning and question-answering benchmarks, including MGSM, MATH-500, CommonSenseQA, and OpenBookQA, into Urdu, collectively referred to as UrduBench, and conduct a comprehensive evaluation of both reasoning-oriented and instruction-tuned LLMs across multiple prompting strategies. Our analysis reveals performance differences across (1) four datasets, (2) five task difficulty levels, (3) diverse model architectures, (4) multiple model scaling settings, and (5) language consistency tests. We find that multi-step and symbolic reasoning tasks pose significant challenges in Urdu, and that stable language alignment is a critical prerequisite for robust reasoning. Overall, our work establishes a scalable methodology for standardized reasoning evaluation in Urdu and provides empirical insights into multilingual reasoning failures. This experimental setup is also broadly applicable to other low-resource languages. The code and datasets will be publicly released.

Figures (7)

• Figure 1: Performance scaling of state-of-the-art LLMs on Urdu benchmark datasets. Average accuracy is the average of MGSM (CoT), MATH-500 (CoT), CommonSenseQA (Direct), and OpenBookQA (Direct) accuracies. Green highlights the top performing model in Urdu.
• Figure 2: Contextual Ensembled Translations Framework with human-in-the-loop
• Figure 3: Performance of different language models across difficulty levels (L1--L5) on the MATH-500-Urdu benchmark. Each subplot shows the accuracy degradation pattern as problem difficulty increases.
• Figure 4: Human preference win--loss comparison between GPT-5.1 refined and human-edited Urdu translations across four datasets. The stacked bars show the number of wins for each translation method, with win rate percentages displayed on the right.
• Figure 5: Evaluation prompt for Urdu MGSM, MATH-500, CommonSenseQA, and OpenBookQA