VerifyBench: Benchmarking Reference-based Reward Systems for Large Language Models

TL;DR

VerifyBench and VerifyBench-Hard provide ground-truth–grounded benchmarks for evaluating reference-based reward verifiers in large language models. The paper differentiates absolute correctness verification from traditional pairwise preference evaluation, and builds two data pipelines with extensive human annotation to assess verifier accuracy. Key findings show state-of-the-art verifiers achieve high accuracy on VerifyBench but struggle on the more difficult VerifyBench-Hard, with reference answers substantially boosting verification performance. The work demonstrates the practical value of high-quality verification signals for RL in reasoning tasks and offers insights into error modes and model-scale effects that guide future improvements.

Abstract

Large reasoning models such as OpenAI o1 and DeepSeek-R1 have achieved remarkable performance in the domain of reasoning. A key component of their training is the incorporation of verifiable rewards within reinforcement learning (RL). However, existing reward benchmarks do not evaluate reference-based reward systems, leaving researchers with limited understanding of the accuracy of verifiers used in RL. In this paper, we introduce two benchmarks, VerifyBench and VerifyBench-Hard, designed to assess the performance of reference-based reward systems. These benchmarks are constructed through meticulous data collection and curation, followed by careful human annotation to ensure high quality. Current models still show considerable room for improvement on both VerifyBench and VerifyBench-Hard, especially smaller-scale models. Furthermore, we conduct a thorough and comprehensive analysis of evaluation results, offering insights for understanding and developing reference-based reward systems. Our proposed benchmarks serve as effective tools for guiding the development of verifier accuracy and the reasoning capabilities of models trained via RL in reasoning tasks.

Figures (14)

• Figure 1: The core distinction between VerifyBench and existing reward benchmarks lambert2025rewardbenchliu2024rmbench is illustrated as follows. Upper panel: Existing reward benchmarks assess the accuracy of a reward system by comparing the ranking of two completions for the same question. Lower panel: In contrast, our proposed VerifyBench evaluates the accuracy of a reward system by determining the correctness of a single completion using a reference answer.
• Figure 2: Overview of the benchmark construction process. The upper section outlines the pipeline used to construct VerifyBench, whereas the lower section details the pipeline for VerifyBench-Hard. The components highlighted by black boxes denote the final entries included in the benchmark.
• Figure 3: The performance(%) of RL across different LLM judges which have various performance on VerifyBench.
• Figure 4: Prompt for answer type classification. {question} and {answer} are placeholders that will be replaced with actual question and answer content.
• Figure 5: Prompt for LLM-as-a-judge evaluation. {question}, {answer}, and {completion} are placeholders that will be replaced with actual question, answer, and completion content.