Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

Qiannian Zhao, Chen Yang, Jinhao Jing, Yunke Zhang, Xuhui Ren, Lu Yu, Shijie Zhang, Hongzhi Yin

TL;DR

The proposed EGPO is a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

Abstract

Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks. In practice, these models are predominantly trained via Reinforcement Learning with Verifiable Rewards (RLVR), yet most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty. We term this discrepancy the uncertainty-reward mismatch, under which high- and low-uncertainty solutions are treated equivalently, preventing the policy from "Know What You Know" and impeding the shift from optimizing for correct answers to optimizing effective reasoning paths. This limitation is especially critical in reasoning-centric tasks such as mathematics and question answering, where performance hinges on the quality of the model's internal reasoning process rather than mere memorization of final answers. To address this, we propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs. EGPO estimates per-sample uncertainty using a zero-overhead entropy proxy derived from token-level likelihoods and aligns it with extrinsic correctness through an asymmetric calibration mechanism that preserves correct reasoning while selectively regulating overconfident failures, thereby enabling stable and uncertainty-aware policy optimization. Moreover, EGPO recovers informative learning signals from otherwise degenerate group-based rollouts without modifying the verifier or reward definition. Extensive experiments across multiple benchmarks demonstrate that the proposed EGPO leads to substantial and consistent improvements in reasoning performance, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning

TL;DR

The proposed EGPO is a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs, establishing a principled path for advancing LRMs through metacognitive entropy calibration.

Abstract

Large reasoning models (LRMs) have emerged as a powerful paradigm for solving complex real-world tasks. In practice, these models are predominantly trained via Reinforcement Learning with Verifiable Rewards (RLVR), yet most existing outcome-only RLVR pipelines rely almost exclusively on a binary correctness signal and largely ignore the model's intrinsic uncertainty. We term this discrepancy the uncertainty-reward mismatch, under which high- and low-uncertainty solutions are treated equivalently, preventing the policy from "Know What You Know" and impeding the shift from optimizing for correct answers to optimizing effective reasoning paths. This limitation is especially critical in reasoning-centric tasks such as mathematics and question answering, where performance hinges on the quality of the model's internal reasoning process rather than mere memorization of final answers. To address this, we propose EGPO, a metacognitive entropy calibration framework that explicitly integrates intrinsic uncertainty into RLVR for enhancing LRMs. EGPO estimates per-sample uncertainty using a zero-overhead entropy proxy derived from token-level likelihoods and aligns it with extrinsic correctness through an asymmetric calibration mechanism that preserves correct reasoning while selectively regulating overconfident failures, thereby enabling stable and uncertainty-aware policy optimization. Moreover, EGPO recovers informative learning signals from otherwise degenerate group-based rollouts without modifying the verifier or reward definition. Extensive experiments across multiple benchmarks demonstrate that the proposed EGPO leads to substantial and consistent improvements in reasoning performance, establishing a principled path for advancing LRMs through metacognitive entropy calibration.
Paper Structure (34 sections, 25 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 34 sections, 25 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Problem Setup
  4. Reinforcement Learning with Verifiable Rewards (RLVR)
  5. Group Relative Policy Optimization (GRPO)
  6. Advantage collapse.
  7. Intrinsic Uncertainty and Entropy Proxy
  8. Methodology
  9. Entropy-Guided Policy Optimization (EGPO)
  10. Calibration mechanism: group-relative entropy weighting
  11. Calibration mechanism: outcome-aware asymmetric calibration
  12. Base advantage, calibrated advantage, and objective
  13. Negative Sample Reinforcement (NSR) for Entirely-Incorrect Groups
  14. Effect under clipping.
  15. Gradient characterization.
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Overview of EGPO: sample-level metacognitive calibration (four quadrants) and group-level rollout triage (entirely-correct / mixed (contains both correct and incorrect outcomes) / entirely-incorrect).
  • Figure 2: Asymmetric calibration in EGPO: correct responses are never down-weighted and incorrect responses are never up-weighted.
  • Figure 3: Density comparison of the entropy on the response for Qwen2.5-Math-1.5B, comparing Base/GRPO/DAPO/EDGE-GRPO/EGPO on correct vs. incorrect rollouts.
  • Figure 4: A qualitative case on Qwen2.5-Math-1.5B comparing Base vs. EGPO, including the entropy on the response.
  • Figure 5: ROC diagnostics for predicting incorrect rollouts using entropy on DeepSeek-R1-Distill-Qwen backbones, comparing thinking-side vs. answer-side entropy.
  • ...and 4 more figures