From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoning

TL;DR

This work systematically probes the reliability of verifiers used in RL with verifiable rewards for mathematical reasoning. It contrasts rule-based verifiers, which are precise but sensitive to answer formatting, with model-based verifiers that better handle diverse answers yet are vulnerable to reward hacking during RL. A hybrid verifier design combines the strengths of both approaches, yielding improved RL performance and data efficiency, though robustness concerns persist across datasets and domains. The study reveals that static verification performance does not always predict RL success and highlights domain-general hacking patterns, underscoring the need for robust verification strategies in scalable reasoning systems.

Abstract

Trustworthy verifiers are essential for the success of reinforcement learning with verifiable reward (RLVR), which is the core methodology behind various large reasoning models such as DeepSeek-R1. In complex domains like mathematical reasoning, rule-based verifiers have been widely adopted in previous works to train strong reasoning models. However, the reliability of these verifiers and their impact on the RL training process remain poorly understood. In this work, we take mathematical reasoning as a case study and conduct a comprehensive analysis of various verifiers in both static evaluation and RL training scenarios. First, we find that current open-source rule-based verifiers often fail to recognize equivalent answers presented in different formats across multiple commonly used mathematical datasets, resulting in non-negligible false negative rates. This limitation adversely affects RL training performance and becomes more pronounced as the policy model gets stronger. Subsequently, we investigate model-based verifiers as a potential solution to address these limitations. While the static evaluation shows that model-based verifiers achieve significantly higher verification accuracy, further analysis and RL results imply that they are highly susceptible to hacking, where they misclassify certain patterns in responses as correct, particularly after fine-tuning. This vulnerability is exploited during policy model optimization, leading to artificially inflated rewards. Our findings underscore the unique challenges inherent to both rule-based and model-based verifiers and provide insights toward developing more accurate and robust reward systems for reinforcement learning.

Figures (12)

• Figure 1: The training and evaluation curves of RL on Qwen-2.5-7B using different verifiers, with the x-axis representing training iterations in all plots. Left illustrates the evaluation accuracy averaged over multiple benchmarks, including GSM8K, MATH500, Minerva Math, OlympiadBench, AIME24, and AMC23. Right depicts changes in reward values during training. The "training rewards" indicate the rewards provided by the corresponding reward system to the policy model, whereas the "oracle rewards" represent rewards the model receives when judged by combining with GPT-4o. All benchmarks are reported with a single sample due to computational constraints; detailed stable results at the peak point are provided in Table \ref{['tab:detailed_perf_rl']}.
• Figure 2: Recall rates of various rule-based verifiers across multiple datasets, evaluated on a subset sampled from Deepseek-R1-Distill-Qwen-32B. “VERL”, “Qwen,” and “HF” refer to the Verl Math Verifier, Qwen-Math Verifier, and Hugging Face Math Verifier, respectively.
• Figure 3: Recall Rate of the Huggingface Math Verifier, evaluated on data sampled from various models across different RL training datasets. "DS" stands for Deepseek, while "Skywork" refers to the Skywork-OR1 dataset.
• Figure 4: Prompt for using GPT-4o as an annotator to provide ground-truth annotations based on the model’s response and the target answer, indicating whether the model’s response aligns with the target answer.
• Figure 5: Examples of correct model responses that are incorrectly flagged as incorrect by the rule-based verifier. upper demonstrates that the model's predicted answer differs from the ground truth only in terms of mathematical formatting, while the lower highlights cases where different representations (such as $\frac{\pi}{4}$ and $45^{o}$) are considered equivalent given the query context (calculating angle $\beta$).