Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards

Xuan Zhang, Ruixiao Li, Zhijian Zhou, Long Li, Yulei Qin, Ke Li, Xing Sun, Xiaoyu Tan, Chao Qu, Yuan Qi

TL;DR

This work tackles the challenge of exploration in reinforcement learning for long-horizon LLM reasoning by introducing MERCI, a count-based intrinsic motivation that uses a lightweight Coin Flip Network (CFN) to estimate state novelty and epistemic uncertainty. By leveraging a simplified Uncertainty Bellman Equation (UBE) for known, deterministic transitions in autoregressive language generation, MERCI converts local reward uncertainty into a principled exploration bonus and integrates it with advanced policy frameworks like GRPO and DAPO. The approach combines a scalable pseudo-count proxy with a structured bonus pipeline—including percentile, spatial coherence, and noise-suppression filtering, followed by trajectory-level normalization and controlled integration—to densify diverse yet coherent reasoning trajectories. Empirical results on challenging math reasoning and SQL generation benchmarks demonstrate consistently improved pass@k and mean@k, indicating that targeted intrinsic motivation can reliably enhance exploration and solution quality in LLM reasoning. Overall, MERCI provides a theoretically grounded, scalable path to richer reasoning in language models with practical implications for improving accuracy and generalization in complex tasks.

Abstract

Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limited exploration, which often drives LLMs toward repetitive and suboptimal reasoning patterns. In this paper, we study the central question of how to design exploration for LLM reasoning and introduce MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards), a novel RL algorithm that augments policy optimization with a principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate the pseudo count and further epistemic uncertainty over reasoning trajectories, and converts them into an intrinsic reward that values novelty while preserving the learning signal from task rewards. We integrate MERCI into some advanced RL frameworks like Group Relative Policy Optimization (GRPO). Experiments on complex reasoning benchmarks demonstrate that MERCI encourages richer and more varied chains of thought, significantly improves performance over strong baselines, and helps the policy escape local routines to discover better solutions. It indicates that our targeted intrinsic motivation can make exploration reliable for language model reasoning.

Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards

TL;DR

This work tackles the challenge of exploration in reinforcement learning for long-horizon LLM reasoning by introducing MERCI, a count-based intrinsic motivation that uses a lightweight Coin Flip Network (CFN) to estimate state novelty and epistemic uncertainty. By leveraging a simplified Uncertainty Bellman Equation (UBE) for known, deterministic transitions in autoregressive language generation, MERCI converts local reward uncertainty into a principled exploration bonus and integrates it with advanced policy frameworks like GRPO and DAPO. The approach combines a scalable pseudo-count proxy with a structured bonus pipeline—including percentile, spatial coherence, and noise-suppression filtering, followed by trajectory-level normalization and controlled integration—to densify diverse yet coherent reasoning trajectories. Empirical results on challenging math reasoning and SQL generation benchmarks demonstrate consistently improved pass@k and mean@k, indicating that targeted intrinsic motivation can reliably enhance exploration and solution quality in LLM reasoning. Overall, MERCI provides a theoretically grounded, scalable path to richer reasoning in language models with practical implications for improving accuracy and generalization in complex tasks.

Abstract

Reinforcement Learning (RL) has become a compelling way to strengthen the multi step reasoning ability of Large Language Models (LLMs). However, prevalent RL paradigms still lean on sparse outcome-based rewards and limited exploration, which often drives LLMs toward repetitive and suboptimal reasoning patterns. In this paper, we study the central question of how to design exploration for LLM reasoning and introduce MERCI (Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards), a novel RL algorithm that augments policy optimization with a principled intrinsic reward. Building on the idea of count-based exploration, MERCI leverages a lightweight Coin Flipping Network (CFN) to estimate the pseudo count and further epistemic uncertainty over reasoning trajectories, and converts them into an intrinsic reward that values novelty while preserving the learning signal from task rewards. We integrate MERCI into some advanced RL frameworks like Group Relative Policy Optimization (GRPO). Experiments on complex reasoning benchmarks demonstrate that MERCI encourages richer and more varied chains of thought, significantly improves performance over strong baselines, and helps the policy escape local routines to discover better solutions. It indicates that our targeted intrinsic motivation can make exploration reliable for language model reasoning.
Paper Structure (51 sections, 1 theorem, 92 equations, 5 figures, 8 tables, 1 algorithm)

This paper contains 51 sections, 1 theorem, 92 equations, 5 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Coin Flip Network (CFN)
  4. Group Relative Policy Optimization (GRPO)
  5. Decouple Clip and Dynamic sAmpling Policy Optimization (DAPO)
  6. Methodology
  7. The Uncertainty Bellman Equation with Known Transitions
  8. Estimate Variance of Reware via CFN
  9. Advantage Estimation
  10. Calculating the Intrinsic Reward from Cumulative Uncertainty
  11. Budget-Aware Exploration Bonus Control
  12. Advantage Normalization and Bonus Integration
  13. Related Work
  14. Reinforcement Learning for LLM Reasoning
  15. Exploration in Reinforcement Learning
  16. ...and 36 more sections

Key Result

Proposition 1

Let $U^h(s,a) \triangleq \mathbb{V}_t[\hat{Q}^{\pi, h}(s, a)]$ be the posterior variance of the Q-value at step $h$, conditioned on the history $\mathcal{F}_t$. Given a known and deterministic transition function, this uncertainty propagates according to the following Bellman equation: where $s'$ is the unique next state reached from $(s,a)$, and $U^{H+1}(\cdot) = 0$.

Figures (5)

  • Figure 1: Overview of the MERCI framework. Two separate networks are used: a policy network $\pi_\theta$ trained with RL, and a CFN network that provides an intrinsic reward. The CFN network, initialized from the same SFT checkpoint $\pi_0$, estimates state novelty to guide the exploration of $\pi_\theta$.
  • Figure 2: The entire pipeline of bonus filtering. Step 1: We rank all tokens within a response by their associated bonus values and retain only those falling within a predefined top percentile (e.g., the top $50\%$ in this figure). Step 2: We only preserve clusters of adjacent tokens that consistently exhibit elevated bonuses (e.g., 3 consecutive tokens in this figure). Step 3: For example, in a math reasoning task without external tools, any Python code potentially generated during LLM rollouts is semantically irrelevant and noisy, so we exclude them from the overall bonus calculation.
  • Figure 3: Some examples of token-level estimated epistemic uncertainty within a response. Red regions indicate relatively higher uncertainty estimates assigned by the CFN to the corresponding token positions, while blue regions indicate relatively lower estimates. The same applies hereafter.
  • Figure 4: In the mathematical reasoning task, a statistical analysis of the occurrence frequency of contiguous token segments within each response that fall within the top $30\%$ of bonus values (after filtering out code-related segments).
  • Figure 5: Validation (i.e., MATH500) accuracy and response length during training.

Theorems & Definitions (1)

  • Proposition 1: Uncertainty Bellman Equation for Known Transitions