Rewarding the Journey, Not Just the Destination: A Composite Path and Answer Self-Scoring Reward Mechanism for Test-Time Reinforcement Learning

TL;DR

The paper tackles the scalability bottleneck of reinforcement learning for large language models by enabling RL on unlabeled data during test time. It introduces COMPASS, a composite reward mechanism consisting of Dual-Calibration Answer Reward (DCAR) and Decisive Path Reward (DPR) that leverage internal signals such as confidence, credibility, and token-level decisiveness to self-score rewards. DCAR stabilizes pseudo-labels via confidence-calibrated self-consistency and credibility, while DPR provides dense per-token feedback to encourage decisive reasoning under high entropy. Empirical results across multiple models and reasoning benchmarks show consistent gains over prior test-time RL methods and demonstrate favorable scaling with model size, indicating a viable path for continuous self-improvement of LLMs without external supervision.

Abstract

Reinforcement Learning (RL) has emerged as a powerful paradigm for advancing Large Language Models (LLMs), achieving remarkable performance in complex reasoning domains such as mathematics and code generation. However, current RL methods face a fundamental scalability bottleneck due to their heavy reliance on human-curated preference data or labeled datasets for reward modeling. To overcome this limitation, we explore RL on unlabeled data where models learn autonomously from continuous experience streams. The core challenge in this setting lies in reliable reward estimation without ground-truth supervision. Existing approaches like Test-Time RL address this through self-consistent consensus, but risk reinforcing incorrect pseudo-labels derived from majority voting. We introduce COMPASS (Composite Path and Answer Self-Scoring), a novel test-time reward mechanism that operates without external supervision. COMPASS integrates two complementary components: the Dual-Calibration Answer Reward (DCAR), which stabilizes training by establishing trustworthy pseudo-labels through confidence and credibility calibration, and the Decisive Path Reward (DPR), which directly optimizes the reasoning process quality beyond mere outcome supervision. By jointly reinforcing trustworthy consensus answers and highly decisive reasoning chains, the COMPASS systematically enhances the model's analytical capabilities. Extensive experiments show that COMPASS achieves significant and consistent performance gains across diverse reasoning tasks and model architectures, advancing a more scalable direction for LLMs to learn from continuous experience.

Figures (7)

• Figure 1: The Composite Path and Answer Self-Scoring (COMPASS) reward mechanism. Given a prompt $q$, the policy LLM samples multiple candidate responses $\{y_1, y_2, \cdots, y_N\}$. The Dual-Calibration Answer Reward (DCAR) firstly constructs a consensus pseudo-label $y^*$ via confidence-calibrated self-consistency, where each response’s contribution is weighted by its decisiveness-derived confidence. DCAR further evaluates the credibility of this consensus by comparing the confidence of the general supporting group against the most confident elite response, yielding a reliability-calibrated answer reward$R_{answer}$. Complementarily, the Decisive Path Reward (DPR) assesses the reasoning process quality by computing step-wise entropy and decisiveness measures to generate the path reward $R_{path}$. Together, DCAR and DPR form a unified intrinsic reward framework that reinforces both trustworthy answers and high-quality reasoning processes.
• Figure 2: Analysis of $\mathcal{PD}_{topk}$ indicators validating confidence-calibrated self-consistency. (a) Distribution of average $\mathcal{PD}_{topk}$ values, showing systematic differences between correct and incorrect responses. (b) Distribution of $\mathcal{PD}_{topk}$ stability throughout generation trajectories, demonstrating more consistent decisiveness in correct answers. (c) Average $\mathcal{PD}_{topk}$ curves across generation steps, revealing that correct responses maintain higher and more stable decisiveness throughout reasoning. These empirical findings confirm that the probability difference between top-1 and top-2 tokens strongly correlates with answer correctness.
• Figure 3: Performance comparison on AIME/AMC/MATH using Qwen2.5-7B.
• Figure 4: Training dynamics comparision on AIME using Qwen2.5-Math-1.5B.
• Figure 5: Ablation Results of COMPASS on AIME using Qwen2.5-Math-1.5B.