Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

Tianyi Xu, Kosei Uemura, Alfred Malengo Kondoro, Tadesse Destaw Belay, Catherine Nana Nyaah Essuman, Ifeoma Okoh, Ganiyat Afolabi, Ayodele Awokoya, David Ifeoluwa Adelani

TL;DR

MGSM-Pro introduces a multilingual extension of GSM-Symbolic to evaluate robust mathematical reasoning across nine languages by generating five digit-varied instantiations per MGSM question. It reveals that numerical instantiations cause large performance drops, especially for low-resource languages, while high-resource languages are comparatively more robust; model robustness varies by architecture and size, with some open models (e.g., GPT-OSS 120B, DeepSeek V3) showing stronger resilience than certain proprietary models. The study also demonstrates that leaderboard rankings are unstable when evaluated over multiple instantiations, arguing for reporting Avg-5 scores as a default to provide a more robust assessment. MGSM-Pro provides a culturally aware, template-based dataset with symbolic and irrelevant-context perturbations, and advocates five-instance evaluations to better reflect real-world math reasoning capabilities. The work has practical implications for benchmarking multilingual mathematical reasoning and informs model development toward greater cross-linguistic robustness.

Abstract

Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that some proprietary models, notably Gemini 2.5 Flash and GPT-4.1, are less robust to digit instantiation, whereas Claude 4.0 Sonnet is more robust. Among open models, GPT-OSS 120B and DeepSeek V3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.

MGSM-Pro: A Simple Strategy for Robust Multilingual Mathematical Reasoning Evaluation

TL;DR

MGSM-Pro introduces a multilingual extension of GSM-Symbolic to evaluate robust mathematical reasoning across nine languages by generating five digit-varied instantiations per MGSM question. It reveals that numerical instantiations cause large performance drops, especially for low-resource languages, while high-resource languages are comparatively more robust; model robustness varies by architecture and size, with some open models (e.g., GPT-OSS 120B, DeepSeek V3) showing stronger resilience than certain proprietary models. The study also demonstrates that leaderboard rankings are unstable when evaluated over multiple instantiations, arguing for reporting Avg-5 scores as a default to provide a more robust assessment. MGSM-Pro provides a culturally aware, template-based dataset with symbolic and irrelevant-context perturbations, and advocates five-instance evaluations to better reflect real-world math reasoning capabilities. The work has practical implications for benchmarking multilingual mathematical reasoning and informs model development toward greater cross-linguistic robustness.

Abstract

Large language models have made substantial progress in mathematical reasoning. However, benchmark development for multilingual evaluation has lagged behind English in both difficulty and recency. Recently, GSM-Symbolic showed a strong evidence of high variance when models are evaluated on different instantiations of the same question; however, the evaluation was conducted only in English. In this paper, we introduce MGSM-Pro, an extension of MGSM dataset with GSM-Symbolic approach. Our dataset provides five instantiations per MGSM question by varying names, digits and irrelevant context. Evaluations across nine languages reveal that many low-resource languages suffer large performance drops when tested on digit instantiations different from those in the original test set. We further find that some proprietary models, notably Gemini 2.5 Flash and GPT-4.1, are less robust to digit instantiation, whereas Claude 4.0 Sonnet is more robust. Among open models, GPT-OSS 120B and DeepSeek V3 show stronger robustness. Based on these findings, we recommend evaluating each problem using at least five digit-varying instantiations to obtain a more robust and realistic assessment of math reasoning.
Paper Structure (38 sections, 5 figures, 9 tables)

This paper contains 38 sections, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Math Reasoning Benchmarks
  4. Robustness in Reasoning
  5. The MGSM-Pro Dataset
  6. The Symbolic Series (SYM)
  7. The Irrelevant Context Series (IC)
  8. Template Construction
  9. Dataset Construction
  10. Experiments
  11. Experiment Setup
  12. Models evaluated
  13. Prompts
  14. Results
  15. Main results
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Relative decrease in accuracy from the original dataset, averaged across the six variants of MGSM-Pro, and aggregated over all nine languages.
  • Figure 2: Workflow diagram illustrating the template creation and final data construction on sample language.
  • Figure 3: Relative Accuracy Decline Across Gemma 3 Family The figure illustrates the relative decline in accuracy for the Gemma-3 family. The drop is measured from the original dataset to two configurations: IC_N# and SYM_N#. Averaged over nine languages.
  • Figure 4: Example of a MGSM-Pro question template alongside its IC sentence template
  • Figure 5: Relative Accuracy Decline Across GPT-OSS Family The figure illustrates the relative decline in accuracy for the GPT-OSS family. The drop is measured from the original dataset to two configurations: IC_N# and SYM_N#. Averaged over nine languages.