Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning

TL;DR

This work addresses the difficulty of improving reasoning in small-scale LLMs by introducing a Confidence-aware Reward Model (C2RM) that penalizes low-confidence correct answers in addition to incorrect ones, aligning both final accuracy and reasoning confidence. The authors construct a specialized training regimen with correctness and confidence labels, train a reward model using contrastive type pairs, and apply PPO-based RL to STEM tasks, showing superior performance over open-source baselines and competitive results with closed models. They validate C2RM through Best-of-N, JudgeBench, and PPO RL evaluations, demonstrating longer, more thorough reasoning and higher accuracy, particularly in complex domains like FOLIO. The approach offers a scalable path for knowledge-intensive post-training under resource constraints and is released with code and model checkpoints to enable broader adoption.

Abstract

Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, numerous technical reports indicate that purely rule-based reward RL frequently results in poor-quality reasoning chains or inconsistencies between reasoning processes and final answers, particularly when the base model is of smaller scale. During the RL exploration process, models might employ low-quality reasoning chains due to the lack of knowledge, occasionally producing correct answers randomly and receiving rewards based on established rule-based judges. This constrains the potential for resource-limited organizations to conduct direct reinforcement learning training on smaller-scale models. We propose a novel confidence-based reward model tailored for enhancing STEM reasoning capabilities. Unlike conventional approaches, our model penalizes not only incorrect answers but also low-confidence correct responses, thereby promoting more robust and logically consistent reasoning. We validate the effectiveness of our approach through static evaluations, Best-of-N inference tests, and PPO-based RL training. Our method outperforms several state-of-the-art open-source reward models across diverse STEM benchmarks. We release our codes and model in https://github.com/qianxiHe147/C2RM.

Figures (6)

• Figure 1: The framework of our reward model data generation and training. We first select high-quality questions where only some of Qwen2.5-72B-Instruct$^\prime$s five answers are correct. Then, for each question, we sample 5 answers (only 3 are shown in the figure for clarity) from each of 3 representative models, label them, and construct training data by treating T&C as positives and others as negatives. Finally, we train our reward model based on Qwen2.5-7B-Instruct.
• Figure 2: Distribution of different pair types across all datasets. These percentages represent the natural proportions of generated responses, and our training data sampling follows these same natural distributions.
• Figure 3: Prompt for closed-source model test in BoN.
• Figure 4: Average accuracy on GPQA-Diamond, MATH500, FOLIO, and MMLU-Pro after PPO training using 30k samples from $\mathcal{Q}$ and $\mathcal{D}$, with different reward models respectively.
• Figure 5: Reward curve of the reward model during PPO training.