Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning

Neeraja Kirtane, Yuvraj Khanna, Peter Relan

TL;DR

MathRobust-LV addresses the gap between strong mathematical performance and robustness to linguistic variation in large language models by introducing a controlled, minimal-perturbation framework that preserves numerical structure and the final answer. The authors generate 520 surface-diverse variants of 130 seed problems (65 from MATH and 65 from AoPS) and evaluate 34 model families in a zero-shot setting, revealing consistent accuracy degradations, especially for smaller models, and only modest gains for frontier models. The key contributions are (i) a scalable methodology for linguistic perturbations that decouples phrasing from formal reasoning, (ii) a large-scale benchmark showing persistent brittleness across architectures and sizes, and (iii) insights into how model scaling and family-specific traits influence robustness. The findings highlight the need for education-focused robustness benchmarks and suggest directions such as automatic rewriters, RL fine-tuning on variant-rich data, and cross-domain extensions to improve reliable math reasoning in real-world tutoring and assessment tools.

Abstract

Large language models excel on math benchmarks, but their math reasoning robustness to linguistic variation is underexplored. While recent work increasingly treats high-difficulty competitions like the IMO as the gold standard for evaluating reasoning, we believe in comprehensive benchmarking of high school-level math problems in real educational settings. We introduce MathRobust-LV, a test set and evaluation methodology that mirrors how instructors rephrase problems across assessments while keeping difficulty constant: we change surface details (names, contexts, variables) while preserving numerical structure and answers. In contrast to prior efforts that alter problem content or emphasize IMO-level tasks, we focus on high-school-level dataset problems at the difficulty level where models are currently deployed in educational settings: tutoring and assessment systems. In these applications, instructors rephrase identical concepts in varied ways, making linguistic robustness essential for reliable deployment. Although MATH data benchmarking is often regarded as saturated, our experiment on 34 models reveals that accuracy declines when moving from the baseline to the variants. These drops are severe for smaller models (9-11%) while stronger models also show measurable degradation. Frontier models like GPT-5, Gemini-2.5pro remain comparatively stable. Our results highlight that robustness to linguistic variation is a fundamental challenge, exposing reasoning vulnerabilities in models.

MathRobust-LV: Evaluation of Large Language Models' Robustness to Linguistic Variations in Mathematical Reasoning

TL;DR

MathRobust-LV addresses the gap between strong mathematical performance and robustness to linguistic variation in large language models by introducing a controlled, minimal-perturbation framework that preserves numerical structure and the final answer. The authors generate 520 surface-diverse variants of 130 seed problems (65 from MATH and 65 from AoPS) and evaluate 34 model families in a zero-shot setting, revealing consistent accuracy degradations, especially for smaller models, and only modest gains for frontier models. The key contributions are (i) a scalable methodology for linguistic perturbations that decouples phrasing from formal reasoning, (ii) a large-scale benchmark showing persistent brittleness across architectures and sizes, and (iii) insights into how model scaling and family-specific traits influence robustness. The findings highlight the need for education-focused robustness benchmarks and suggest directions such as automatic rewriters, RL fine-tuning on variant-rich data, and cross-domain extensions to improve reliable math reasoning in real-world tutoring and assessment tools.

Abstract

Large language models excel on math benchmarks, but their math reasoning robustness to linguistic variation is underexplored. While recent work increasingly treats high-difficulty competitions like the IMO as the gold standard for evaluating reasoning, we believe in comprehensive benchmarking of high school-level math problems in real educational settings. We introduce MathRobust-LV, a test set and evaluation methodology that mirrors how instructors rephrase problems across assessments while keeping difficulty constant: we change surface details (names, contexts, variables) while preserving numerical structure and answers. In contrast to prior efforts that alter problem content or emphasize IMO-level tasks, we focus on high-school-level dataset problems at the difficulty level where models are currently deployed in educational settings: tutoring and assessment systems. In these applications, instructors rephrase identical concepts in varied ways, making linguistic robustness essential for reliable deployment. Although MATH data benchmarking is often regarded as saturated, our experiment on 34 models reveals that accuracy declines when moving from the baseline to the variants. These drops are severe for smaller models (9-11%) while stronger models also show measurable degradation. Frontier models like GPT-5, Gemini-2.5pro remain comparatively stable. Our results highlight that robustness to linguistic variation is a fundamental challenge, exposing reasoning vulnerabilities in models.

Paper Structure

This paper contains 21 sections, 4 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. How our work differs.
  4. Methodology Design
  5. Seed Dataset Selection
  6. Variation Generation
  7. Experimental Setup
  8. Results
  9. Preliminary Results
  10. Results on MathRobust-LV data
  11. General Trends
  12. Scaling Effects
  13. Architectural Differences
  14. Closed source models
  15. Additional Experiments and Observations
  16. ...and 6 more sections

Figures (4)

  • Figure 1: All the different types of substitutions done in a question to generate variations. Change are highlighted in different colors to show the different components that are changed.
  • Figure 2: Preliminary experiment to check accuracy drops on the GSM symbolic dataset (GSM Main) as compared to the test set of GSM8k (GSM1k). More details are in Table \ref{['tab:gsm_model_performance']}.
  • Figure 3: Accuracy drops for some notable models on our variation data.
  • Figure 4: Illustrative example where a model gets the answer of the original question correct but fails on the variation. The two problems are mathematically equivalent (only surface wording differs). The model succeeds on the original but fails on the variant, illustrating brittleness under benign rephrasings.