Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model

Prasanna Mayilvahanan, Ricardo Dominguez-Olmedo, Thaddäus Wiedemer, Wieland Brendel

TL;DR

MATH-Beyond (MATH-B) introduces a zero-baseline benchmark to assess whether RL finetuning can genuinely expand a base model's reasoning beyond its pretraining capabilities, rather than merely sharpening existing solutions. The authors formalize an evaluation framework using the empirical pass@k metric and decompose performance into Expansion, Shrinkage, Preservation, and Consolidation to diagnose where gains arise. Their construction pipeline pools problems from DAPO-Math-17K and DeepScaleR, applies stringent quality filters, verifies ground-truth answers with frontier models, and performs pass@1024 evaluations across diverse base and supplementary models, ultimately yielding the MATH-B-U union set and MATH-B-I intersection set. Empirically, current RL methods offer only modest expansion on MATH-B, while long-Chain-of-Thought distillation shows substantial gains, highlighting the need for exploration-driven methods to achieve genuine boundary expansion and motivating future research toward novel reasoning pathways.

Abstract

With the advent of DeepSeek-R1, a new wave of reinforcement learning (RL) methods has emerged that seem to unlock stronger mathematical reasoning. However, a closer look at the open-source ecosystem reveals a critical limitation: with sufficiently many draws (e.g., $\texttt{pass@1024}$), many existing base models already solve nearly all questions on widely used math benchmarks such as MATH-500 and AIME 2024. This suggests that the RL fine-tuning methods prevalent in the LLM reasoning literature largely sharpen existing solution modes rather than discovering entirely new ones. Such sharpening stands in contrast to the broader promise of RL: to foster exploration and to acquire new skills. To move beyond this plateau, we introduce MATH-Beyond (MATH-B), a benchmark deliberately constructed to defeat common open-source models of up to 8B parameters even under large sampling budgets. Improving performance on our benchmark via RL requires methods that learn to reason in ways that go beyond base model capabilities in repeated sampling. Since the problems are drawn from subsets of DAPO-Math-17K and DeepScaleR datasets, they remain topically equivalent to standard high-school math. Validating our premise, RL fine-tuned models such as Nemotron-Research-Reasoning-Qwen-1.5B and DeepScaleR-1.5B-Preview perform poorly on MATH-B at $\texttt{pass@1024}$, showing how existing approaches fall short on tackling harder instances. We hope MATH-B will catalyze exploration-driven RL approaches that elicit deeper reasoning capabilities. We release MATH-B at https://huggingface.co/datasets/brendel-group/MATH-Beyond.

MATH-Beyond: A Benchmark for RL to Expand Beyond the Base Model

TL;DR

MATH-Beyond (MATH-B) introduces a zero-baseline benchmark to assess whether RL finetuning can genuinely expand a base model's reasoning beyond its pretraining capabilities, rather than merely sharpening existing solutions. The authors formalize an evaluation framework using the empirical pass@k metric and decompose performance into Expansion, Shrinkage, Preservation, and Consolidation to diagnose where gains arise. Their construction pipeline pools problems from DAPO-Math-17K and DeepScaleR, applies stringent quality filters, verifies ground-truth answers with frontier models, and performs pass@1024 evaluations across diverse base and supplementary models, ultimately yielding the MATH-B-U union set and MATH-B-I intersection set. Empirically, current RL methods offer only modest expansion on MATH-B, while long-Chain-of-Thought distillation shows substantial gains, highlighting the need for exploration-driven methods to achieve genuine boundary expansion and motivating future research toward novel reasoning pathways.

Abstract

With the advent of DeepSeek-R1, a new wave of reinforcement learning (RL) methods has emerged that seem to unlock stronger mathematical reasoning. However, a closer look at the open-source ecosystem reveals a critical limitation: with sufficiently many draws (e.g., ), many existing base models already solve nearly all questions on widely used math benchmarks such as MATH-500 and AIME 2024. This suggests that the RL fine-tuning methods prevalent in the LLM reasoning literature largely sharpen existing solution modes rather than discovering entirely new ones. Such sharpening stands in contrast to the broader promise of RL: to foster exploration and to acquire new skills. To move beyond this plateau, we introduce MATH-Beyond (MATH-B), a benchmark deliberately constructed to defeat common open-source models of up to 8B parameters even under large sampling budgets. Improving performance on our benchmark via RL requires methods that learn to reason in ways that go beyond base model capabilities in repeated sampling. Since the problems are drawn from subsets of DAPO-Math-17K and DeepScaleR datasets, they remain topically equivalent to standard high-school math. Validating our premise, RL fine-tuned models such as Nemotron-Research-Reasoning-Qwen-1.5B and DeepScaleR-1.5B-Preview perform poorly on MATH-B at , showing how existing approaches fall short on tackling harder instances. We hope MATH-B will catalyze exploration-driven RL approaches that elicit deeper reasoning capabilities. We release MATH-B at https://huggingface.co/datasets/brendel-group/MATH-Beyond.

Paper Structure

This paper contains 43 sections, 10 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Framework for Evaluating Model Expansion
  3. The Empirical pass@k Metric
  4. Decomposition and Key Metrics
  5. Expansion (Primary Metric).
  6. Shrinkage.
  7. Preservation.
  8. Consolidation.
  9. Interpretation.
  10. Special Case: The MATH-B Benchmark
  11. Creating the benchmark
  12. On the Pitfalls of Verification
  13. MATH-B Construction Pipeline
  14. Base Dataset Sourcing
  15. Quality filters
  16. ...and 28 more sections

Figures (7)

  • Figure 1: MATH-Beyond: Benchmark Construction and Difficulty.Left: Schematic of the MATH-B creation process. A large set of problems from DAPO-Math-17K and DeepScaleR is first refined through quality filters to ensure answer correctness and verifiability. This is followed by evaluation against a gauntlet of open-source base models ($\leq$ 8B, e.g., Qwen3, Qwen2.5 (-Math), DeepSeek-R1-Distill) at a pass@1024 budget to isolate problems that lie beyond their limits. The filtering yields the MATH-B suite of benchmarks: a 41-problem intersection set (unsolved by all base models) for evaluating universal difficulty, and a larger 181-problem union set (unsolved by at least one model) with model-specific splits for targeted analysis. This suite provides a rigorous testbed to drive the development of exploration methods for RL. Right: An illustration of MATH-B's significant difficulty compared to common test sets like AIME24. Representative open-source models like Qwen2.5 achieve near-zero pass@1024 scores on MATH-B, highlighting its difficulty. Qwen2.5 results are from yue2025doesreinforcementlearningreally.
  • Figure 2: Characteristics of the MATH-B-U dataset. Left subplot shows the distribution of math domains. Right subplot show the distribution of source datasets.
  • Figure 3: Difficulty distribution. The left subplot shows the difficulty distribution of the MATH-Beyond-Union set, while the right subplot shows that of the MATH-Beyond-Intersection set. The wide spread of difficulty levels highlights a key mismatch: the problems that models find challenging are not necessarily those that humans typically struggle with.
  • Figure 4: Evolution of Expansion Rate for RL Models. Models are evaluated on the MATH-B problems failed by their respective base models (115 for R1-Qwen2.5-1.5B; 99 for R1-Qwen2.5-7B).
  • Figure 5: Distribution of ground truth (final-answers) in MATH-B-U. We use the log-scale for better readability.
  • ...and 2 more figures