Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning

TL;DR

This work tackles the limitations of test-time reinforcement learning that relies on majority voting by introducing SCOPE, a framework that uses average step confidence and subgroup-specific consensus to produce denser, more reliable reward signals. By partitioning candidate outputs into subgroups and using Pareto optimization to balance quality and exploration, SCOPE provides nuanced credit assignment and diverse supervision targets. Empirical results across multiple models and benchmarks show consistent improvements over strong baselines, including notable gains on AIME and AMC datasets, with ablations confirming the necessity of both step-wise confidence and subgroup partitioning. The approach advances unsupervised RL for LLM reasoning and offers practical improvements for fine-grained, reliable test-time learning.

Abstract

Test-time reinforcement learning mitigates the reliance on annotated data by using majority voting results as pseudo-labels, emerging as a complementary direction to reinforcement learning with verifiable rewards (RLVR) for improving reasoning ability of large language models (LLMs). However, this voting strategy often induces confirmation bias and suffers from sparse rewards, limiting the overall performance. In this work, we propose subgroup-specific step-wise confidence-weighted pseudo-label estimation (SCOPE), a framework integrating model confidence and dynamic subgroup partitioning to address these issues. Specifically, SCOPE integrates the proposed step-wise confidence into pseudo label deduction, prioritizing high-quality reasoning paths over simple frequency count. Furthermore, it dynamically partitions the candidate outputs pool into independent subgroups by balancing reasoning quality against exploration diversity. By deriving local consensus via repeat sampling for each sub group, SCOPE provides diverse supervision targets to encourage broader exploration. We conduct experiments across various models and benchmarks, experimental results show that SCOPE consistently outperforms recent baselines. Notably, SCOPE achieving relative improvements of 13.1\% on challenging AIME 2025 and 8.1\% on AMC. The code is released at \href{https://github.com/szu-tera/SCOPE}{https://github.com/szu-tera/SCOPE}.

Figures (9)

• Figure 1: Illustration of the difference between TTRLzuo2025ttrl and our method. Top: consensus label estimation with step-wise confidence weighting. Bottom: group partition and reward calculation using subgroup-specific consensus labels.
• Figure 2: Overview of the SCOPE framework. The process involves (a) generating responses with step-wise confidence, (b) estimating consensus labels via weighted voting, (c) evaluating different subgroup partitions, (d) employing Pareto optimization to select the optimal subgroup size $m^*$ by balancing (g) quality and exploration metrics, and (e) computing rewards using the optimized subgroup strategy for model updates.
• Figure 3: Analysis of the trade-off parameter $\lambda$.
• Figure 4: Impact analysis of confidence granularity. Comparison between SCOPE with our proposed step-wise confidence and alternative aggregation strategies.
• Figure 5: Impact analysis of subgroup size on training dynamics. Comparison between SCOPE with automatic subgroup size selection and variants with fixed subgroup sizes.