Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR

Hao Yi, Yulan Hu, Xin Li, Sheng Ouyang, Lizhong Ding, Yong Liu

TL;DR

This paper tackles the high labeling cost of reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning by introducing uncertainty-consistency guided active learning. It defines offline and online metrics that align subjective model uncertainty with objective correctness using the point-biserial correlation and a normalized online variant, and demonstrates that classic uncertainty-based AL fails in this setting. The proposed approach enables training with as little as 30% of the data while achieving or surpassing full-data performance on multiple LLMs and reasoning tasks, through principled sample selection that stabilizes gradients and preserves exploration. Overall, uncertainty-consistency provides a scalable, data-efficient framework for RLVR that significantly reduces annotation costs without sacrificing reasoning accuracy, facilitating practical deployment of reasoning-enhanced LLMs.

Abstract

Large Language Models (LLMs) have recently improved mathematical reasoning through Reinforcement Learning with Verifiable Reward (RLVR). However, existing RLVR algorithms require large query budgets, making annotation costly. We investigate whether fewer but more informative queries can yield similar or superior performance, introducing active learning (AL) into RLVR. We identify that classic AL sampling strategies fail to outperform random selection in this setting, due to ignoring objective uncertainty when only selecting by subjective uncertainty. This work proposes an uncertainty consistency metric to evaluate how well subjective uncertainty aligns with objective uncertainty. In the offline setting, this alignment is measured using the Point-Biserial Correlation Coefficient (PBC). For online training, because of limited sampling and dynamically shifting output distributions, PBC estimation is difficult. Therefore, we introduce a new online variant, computed from normalized advantage and subjective uncertainty. Theoretically, we prove that the online variant is strictly negatively correlated with offline PBC and supports better sample selection. Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30% of the data, effectively reducing the cost of RLVR for reasoning tasks.

Learn More with Less: Uncertainty Consistency Guided Query Selection for RLVR

TL;DR

This paper tackles the high labeling cost of reinforcement learning with verifiable rewards (RLVR) for mathematical reasoning by introducing uncertainty-consistency guided active learning. It defines offline and online metrics that align subjective model uncertainty with objective correctness using the point-biserial correlation and a normalized online variant, and demonstrates that classic uncertainty-based AL fails in this setting. The proposed approach enables training with as little as 30% of the data while achieving or surpassing full-data performance on multiple LLMs and reasoning tasks, through principled sample selection that stabilizes gradients and preserves exploration. Overall, uncertainty-consistency provides a scalable, data-efficient framework for RLVR that significantly reduces annotation costs without sacrificing reasoning accuracy, facilitating practical deployment of reasoning-enhanced LLMs.

Abstract

Large Language Models (LLMs) have recently improved mathematical reasoning through Reinforcement Learning with Verifiable Reward (RLVR). However, existing RLVR algorithms require large query budgets, making annotation costly. We investigate whether fewer but more informative queries can yield similar or superior performance, introducing active learning (AL) into RLVR. We identify that classic AL sampling strategies fail to outperform random selection in this setting, due to ignoring objective uncertainty when only selecting by subjective uncertainty. This work proposes an uncertainty consistency metric to evaluate how well subjective uncertainty aligns with objective uncertainty. In the offline setting, this alignment is measured using the Point-Biserial Correlation Coefficient (PBC). For online training, because of limited sampling and dynamically shifting output distributions, PBC estimation is difficult. Therefore, we introduce a new online variant, computed from normalized advantage and subjective uncertainty. Theoretically, we prove that the online variant is strictly negatively correlated with offline PBC and supports better sample selection. Experiments show our method consistently outperforms random and classic AL baselines, achieving full-dataset performance while training on only 30% of the data, effectively reducing the cost of RLVR for reasoning tasks.
Paper Structure (22 sections, 2 theorems, 20 equations, 4 figures, 5 tables)

This paper contains 22 sections, 2 theorems, 20 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminary
  4. Reinforcement Learning based on Verifiable Rewards (RLVR)
  5. Uncertainty Estimation
  6. Method
  7. Uncertainty Consistency in offline scenarios
  8. Uncertainty Consistency in online scenarios
  9. Experiment
  10. SetUp
  11. Main Result
  12. Ablation Study
  13. Discussion
  14. Conclusion
  15. Appendix
  16. ...and 7 more sections

Key Result

Theorem 1

For the same model $\pi_{\theta}$, the covariance between $r_{pb}$ and $r_{pb}^{\text{online}}$ is less than zero, i.e., $\mathrm{Cov}(r_{pb}, r_{pb}^{\text{online}}) < 0$.

Figures (4)

  • Figure 1: (\ref{['fig:gradient_norm']}) Gradient norm dynamics for inconsistent vs. consistent samples. (\ref{['fig:online-offline-corr']}) Correlation between online and offline uncertainty consistency metrics.
  • Figure 2: Offline (left) and online (right) query selection procedures.
  • Figure 3: Ablation study and discussion on Qwen2.5-7B: (1-a)&(1-b) Choosing Top 30% or Bottom 30% $r_{pb}$ samples. (1-c)&(1-d) Sensitivity of $\gamma$ in Equation \ref{['equ:online']}. (2-a)&(2-b) Responses length and entropy during online training. (2-c)&(2-d) Different Sample Ratios.
  • Figure 4: Gradient inner product heatmap.

Theorems & Definitions (4)

  • Theorem 1: Negative Correlation between $r_{pb}$ and $r_{pb}^{\text{online}}$
  • Theorem 2: Equivalent between Maximizing Decrease in Sample Uncertainty and Maximizing $r_{pb}^{online}$
  • proof
  • proof