Curing Miracle Steps in LLM Mathematical Reasoning with Rubric Rewards

TL;DR

This paper investigates why outcome-based rewards in LLMs lead to false positives in mathematical reasoning, exemplified by Miracle Steps and memorization. It introduces the Rubric Reward Model (RRM), a process-oriented reward that scores entire reasoning traces against problem-specific rubrics to penalize flawed logic. Across four math benchmarks, RRM-based reinforcement learning yields substantial gains, especially under verification (Verified Pass@N), and dramatically reduces Miracle Steps, demonstrating that rewarding reasoning quality improves both accuracy and reliability. The work highlights practical implications for building trustworthy mathematical reasoning systems and outlines limitations and avenues for automation and adaptation across domains.

Abstract

Large language models for mathematical reasoning are typically trained with outcome-based rewards, which credit only the final answer. In our experiments, we observe that this paradigm is highly susceptible to reward hacking, leading to a substantial overestimation of a model's reasoning ability. This is evidenced by a high incidence of false positives - solutions that reach the correct final answer through an unsound reasoning process. Through a systematic analysis with human verification, we establish a taxonomy of these failure modes, identifying patterns like Miracle Steps - abrupt jumps to a correct output without a valid preceding derivation. Probing experiments suggest a strong association between these Miracle Steps and memorization, where the model appears to recall the answer directly rather than deriving it. To mitigate this systemic issue, we introduce the Rubric Reward Model (RRM), a process-oriented reward function that evaluates the entire reasoning trajectory against problem-specific rubrics. The generative RRM provides fine-grained, calibrated rewards (0-1) that explicitly penalize logical flaws and encourage rigorous deduction. When integrated into a reinforcement learning pipeline, RRM-based training consistently outperforms outcome-only supervision across four math benchmarks. Notably, it boosts Verified Pass@1024 on AIME2024 from 26.7% to 62.6% and reduces the incidence of Miracle Steps by 71%. Our work demonstrates that rewarding the solution process is crucial for building models that are not only more accurate but also more reliable.

Figures (11)

• Figure 1: The Standard Pass@N and Verified Pass@N on AIME2024 for a Qwen3-4B-Base model trained with outcome-based reward (i.e. Qwen3-4B-Outcome).
• Figure 2: (a) Illustration of the direct answering setting. (b) In the direct answering setting, we report the proportion of samples from four mathematical reasoning datasets where Qwen3-4B-Outcome’s answers fall within the Top-k candidates (beam search). (c) Comparison between Miracle Steps false positive samples and other types of false positive samples.
• Figure 3: (a) Performance comparison of three methods for identifying false positive samples. (b) False positive rates across different rubric reward ranges.
• Figure 4: The pipeline of constructing our rubric reward model.
• Figure 5: SFT vs. RL RRM. Accuracy: score deviation from Gemini's score; Stability: maximum variation across $5$ runs, temperature set to 1.0.