RIDE: Difficulty Evolving Perturbation with Item Response Theory for Mathematical Reasoning

TL;DR

RIDE tackles the reliability of mathematical reasoning in LLMs by applying Item Response Theory to quantify question difficulty across a large set of simulated learners. It then uses a pairwise difficulty ranker and GSPO-based reinforcement learning to rewrite questions, evolving them to intrinsically higher difficulty while maintaining solvability. The approach yields perturbed benchmarks (RIDE-AMC, RIDE-AIME) with substantial performance drops across frontier models, demonstrating enhanced robustness assessment and potential for data augmentation. The work provides a scalable framework for difficulty-driven adversarial evaluation and releases substantial resources to support future research in robust mathematical reasoning.

Abstract

Large language models (LLMs) achieve high performance on mathematical reasoning, but these results can be inflated by training data leakage or superficial pattern matching rather than genuine reasoning. To this end, an adversarial perturbation-based evaluation is needed to measure true mathematical reasoning ability. Current rule-based perturbation methods often generate ill-posed questions and impede the systematic evaluation of question difficulty and the evolution of benchmarks. To bridge this gap, we propose RIDE, a novel adversarial question-rewriting framework that leverages Item Response Theory (IRT) to rigorously measure question difficulty and to generate intrinsically more challenging, well-posed variations of mathematical problems. We employ 35 LLMs to simulate students and build a difficulty ranker from their responses. This ranker provides a reward signal during reinforcement learning and guides a question-rewriting model to reformulate existing questions across difficulty levels. Applying RIDE to competition-level mathematical benchmarks yields perturbed versions that degrade advanced LLM performance, with experiments showing an average 21.73% drop across 26 models, thereby exposing limited robustness in mathematical reasoning and confirming the validity of our evaluation approach.

Figures (9)

• Figure 1: Answers of Qwen3-30B-A3B to a mathematical question and variation rewritten by RIDE.
• Figure 2: The overall architecture of RIDE.
• Figure 3: Case study for our method. The original question is sampled from DeepMath dataset, and rewritten question is the corresponding rewritten version in RIDE-DeepMath.
• Figure 4: Comparison of pass@n performance before and after benchmark perturbation.
• Figure 5: Statics of average token count, average inference time, correct rate and cosine similarity across different models.