Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Confidence as a Reward: Transforming LLMs into Reward Models

He Du, Bowen Li, Chengxing Xie, Chang Gao, Kai Chen, Dacheng Tao

TL;DR

The paper tackles the cost and data requirements of reward modeling for LLMs by proposing Confidence-as-a-Reward (CRew), a training-free reward proxy that uses the mean probability of final-answer tokens to evaluate solutions, particularly for close-ended math tasks. It theoretically justifies confidence as a reward via a formal set of conditions and demonstrates strong empirical performance on MATH500 and RewardMATH, outperforming several training-free baselines and matching many trained reward models. The authors introduce CRew-DPO, a self-training strategy that builds preference data from confidence and correctness and uses DPO to improve evaluation capabilities, achieving substantial gains and generalizing across tasks. Together, CRew and CRew-DPO offer a data- and compute-efficient path to robust reward modeling and model evaluation with strong practical impact for scalable, automatic alignment and data filtering.

Abstract

Reward models can significantly enhance the reasoning capabilities of large language models (LLMs), but they typically require extensive curated data and costly training. To mitigate these challenges, training-free approaches such as LLM-as-a-Judge leverage the intrinsic reasoning abilities of LLMs to evaluate responses, achieving promising results. Recent works have also indicated that model confidence can serve effectively as a reward metric, distinguishing between chain-of-thought (CoT) and non-CoT paths. However, the concept of using confidence as a reward has not been comprehensively studied. In this work, we systematically investigate Confidence-as-a-Reward (CRew), a simple yet powerful training-free method that utilizes token-level confidence in the model's final answers as a proxy for reward, especially suitable for close-ended tasks. Through extensive experiments on mathematical reasoning tasks, we demonstrate that CRew outperforms existing training-free reward approaches on the MATH500 and RewardMATH benchmarks, and even surpasses most trained reward models. We further identify a strong correlation between CRew scores and the actual reasoning performance of the model. Additionally, we find that CRew can effectively filter high-quality training data. Building upon these insights, we propose CRew-DPO, a training strategy that constructs preference data from confidence scores combined with correctness signals. Finetuning with CRew-DPO further enhances the model's judging capabilities and consistently outperforms existing self-training methods.

Confidence as a Reward: Transforming LLMs into Reward Models

TL;DR

The paper tackles the cost and data requirements of reward modeling for LLMs by proposing Confidence-as-a-Reward (CRew), a training-free reward proxy that uses the mean probability of final-answer tokens to evaluate solutions, particularly for close-ended math tasks. It theoretically justifies confidence as a reward via a formal set of conditions and demonstrates strong empirical performance on MATH500 and RewardMATH, outperforming several training-free baselines and matching many trained reward models. The authors introduce CRew-DPO, a self-training strategy that builds preference data from confidence and correctness and uses DPO to improve evaluation capabilities, achieving substantial gains and generalizing across tasks. Together, CRew and CRew-DPO offer a data- and compute-efficient path to robust reward modeling and model evaluation with strong practical impact for scalable, automatic alignment and data filtering.

Abstract

Reward models can significantly enhance the reasoning capabilities of large language models (LLMs), but they typically require extensive curated data and costly training. To mitigate these challenges, training-free approaches such as LLM-as-a-Judge leverage the intrinsic reasoning abilities of LLMs to evaluate responses, achieving promising results. Recent works have also indicated that model confidence can serve effectively as a reward metric, distinguishing between chain-of-thought (CoT) and non-CoT paths. However, the concept of using confidence as a reward has not been comprehensively studied. In this work, we systematically investigate Confidence-as-a-Reward (CRew), a simple yet powerful training-free method that utilizes token-level confidence in the model's final answers as a proxy for reward, especially suitable for close-ended tasks. Through extensive experiments on mathematical reasoning tasks, we demonstrate that CRew outperforms existing training-free reward approaches on the MATH500 and RewardMATH benchmarks, and even surpasses most trained reward models. We further identify a strong correlation between CRew scores and the actual reasoning performance of the model. Additionally, we find that CRew can effectively filter high-quality training data. Building upon these insights, we propose CRew-DPO, a training strategy that constructs preference data from confidence scores combined with correctness signals. Finetuning with CRew-DPO further enhances the model's judging capabilities and consistently outperforms existing self-training methods.
Paper Structure (26 sections, 11 equations, 4 figures, 8 tables)

This paper contains 26 sections, 11 equations, 4 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. CRew
  5. What is confidence?
  6. Why confidence can be used as a reward?
  7. CRew-DPO
  8. Experiments with CRew
  9. Setup
  10. Datasets
  11. CRew as a Strong Reward Proxy
  12. Stronger Models are Better Evaluators
  13. CRew as a Data Filtering Stretegy
  14. Experiments with CRew-DPO
  15. Setup
  16. ...and 11 more sections

Figures (4)

  • Figure 1: An Overview of Our Approach. The upper part illustrates how to calculate the confidence reward of a given solution. The bottom part describes how training on close-end tasks can lead to the self-improvement of the model’s evaluation ability. For a given question, the model outputs multiple solutions, each solution consists of a rationale and an answer. The answers are extracted from all the solutions to calculate the confidence. Finally, the top-K pairs with the largest confidence difference among all correct and incorrect solution pairs are selected for DPO training.
  • Figure 2: The correlation between mathematical reasoning ability and evaluation ability. Using eight models with varying capabilities from the Llama-3 and Qwen-2.5 families, the results demonstrate a strong positive correlation, with a correlation coefficient reaches 0.83. This demonstrates the consistency between mathematical reasoning ability and mathematical evaluation ability.
  • Figure 3: The results of the Llama3-8B-Base model on MATH500 using the data with the highest and lowest confidence (calculated by three different critic models.) versus the original training set, all under the same training settings.
  • Figure 4: A comparison between the confidence reward model trained on MATH, the confidence reward model trained on GSM8K, and the untrained model itself. The two subplots on the left and right respectively show the performance of the actor model on MATH and GSM8K under the guidance of different reward models. The testing method here is the same as described earlier, i.e., SC+Confidence, with a temperature of 1.0 and 16 solutions sampled for each question. Self confidence means using the untrained actor itself to calculate the confidence.