Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning

Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam, Swakkhar Shatabda

TL;DR

This work introduces MathMist, a parallel multilingual benchmark for mathematical problem solving and reasoning across seven languages. The dataset includes over 21K aligned artifacts, with 10.1K parallel problems, 2.2K MCQs, and 8.6K perturbed solutions, plus code-switched and perturbation-based evaluation capabilities. Evaluations of diverse models under zero-shot and chain-of-thought prompting reveal persistent cross-lingual reasoning deficiencies, especially in low-resource languages. The paper also details a full data creation pipeline and an LLM-based evaluation framework, offering a valuable resource for advancing equitable, multilingual mathematical reasoning in AI systems.

Abstract

Mathematical reasoning remains one of the most challenging domains for large language models (LLMs), requiring not only linguistic understanding but also structured logical deduction and numerical precision. While recent LLMs demonstrate strong general-purpose reasoning abilities, their mathematical competence across diverse languages remains underexplored. Existing benchmarks primarily focus on English or a narrow subset of high-resource languages, leaving significant gaps in assessing multilingual and cross-lingual mathematical reasoning. To address this, we introduce MathMist, a parallel multilingual benchmark for mathematical problem solving and reasoning. MathMist encompasses over 21K aligned question-answer pairs across seven languages, representing a balanced coverage of high-, medium-, and low-resource linguistic settings. The dataset captures linguistic variety, multiple types of problem settings, and solution synthesizing capabilities. We systematically evaluate a diverse suite of models, including open-source small and medium LLMs, proprietary systems, and multilingual-reasoning-focused models, under zero-shot, chain-of-thought (CoT), and code-switched reasoning paradigms. Our results reveal persistent deficiencies in LLMs' ability to perform consistent and interpretable mathematical reasoning across languages, with pronounced degradation in low-resource settings. All the codes and data are available at GitHub: https://github.com/mahbubhimel/MathMist

MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning

TL;DR

This work introduces MathMist, a parallel multilingual benchmark for mathematical problem solving and reasoning across seven languages. The dataset includes over 21K aligned artifacts, with 10.1K parallel problems, 2.2K MCQs, and 8.6K perturbed solutions, plus code-switched and perturbation-based evaluation capabilities. Evaluations of diverse models under zero-shot and chain-of-thought prompting reveal persistent cross-lingual reasoning deficiencies, especially in low-resource languages. The paper also details a full data creation pipeline and an LLM-based evaluation framework, offering a valuable resource for advancing equitable, multilingual mathematical reasoning in AI systems.

Abstract

Mathematical reasoning remains one of the most challenging domains for large language models (LLMs), requiring not only linguistic understanding but also structured logical deduction and numerical precision. While recent LLMs demonstrate strong general-purpose reasoning abilities, their mathematical competence across diverse languages remains underexplored. Existing benchmarks primarily focus on English or a narrow subset of high-resource languages, leaving significant gaps in assessing multilingual and cross-lingual mathematical reasoning. To address this, we introduce MathMist, a parallel multilingual benchmark for mathematical problem solving and reasoning. MathMist encompasses over 21K aligned question-answer pairs across seven languages, representing a balanced coverage of high-, medium-, and low-resource linguistic settings. The dataset captures linguistic variety, multiple types of problem settings, and solution synthesizing capabilities. We systematically evaluate a diverse suite of models, including open-source small and medium LLMs, proprietary systems, and multilingual-reasoning-focused models, under zero-shot, chain-of-thought (CoT), and code-switched reasoning paradigms. Our results reveal persistent deficiencies in LLMs' ability to perform consistent and interpretable mathematical reasoning across languages, with pronounced degradation in low-resource settings. All the codes and data are available at GitHub: https://github.com/mahbubhimel/MathMist
Paper Structure (37 sections, 1 equation, 3 figures, 15 tables)

This paper contains 37 sections, 1 equation, 3 figures, 15 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-Resource Math Datasets.
  4. Multilingual Mathematical Reasoning.
  5. Corpus Creation
  6. Data Sourcing
  7. Multiple-Choice Questions Generation
  8. Perturbation Generation
  9. Multilingual Translation Pipeline
  10. Corpus Statistics
  11. Experimental Setup
  12. Models
  13. Evaluation Metrics
  14. Accuracy.
  15. LLM-as-a-Judge.
  16. ...and 22 more sections

Figures (3)

  • Figure 1: Overview of MathMist data creation and evaluation pipeline. (Left) Data Sourcing and corpus creation uses Gemini OCR on textbooks, stores data to JSONL, and applies human verification. (Center) Synthetic data generation encompasses Multiple Choice Question (MCQ) generation, Cross-Lingual Translation, and Solution Perturbation. (Right) The evaluation process tests various LLMs under different prompt settings.
  • Figure 2: Mathematical reasoning performance across seven typologically diverse languages. Models with broad multilingual pretraining achieve higher accuracy, while specialized models show greater cross-lingual variability.
  • Figure 3: Comparison of the quantitative performance of code-switching between high$\leftrightarrow$low resource languages. In this context, BN stands for Bangla, EN refers to English, and Q denotes question.