What is the objective of reasoning with reinforcement learning?

TL;DR

The paper reframes RL fine-tuning of large language models with binary rewards as stochastic gradient ascent on a monotone transform $J_h(\theta)=\mathbb{E}_{x\sim Q}[h(p_\theta(C|x))]$ of the probability of correctness $p_\theta(C|x)$, unifying vanilla REINFORCE, rejection sampling, and GRPO under a single perspective. It derives explicit induced loss functions $h_M$ for broad reweighting schemes via a Bernstein polynomial basis, showing that rejection sampling targets $h(t) \approx \log t$ while GRPO targets $h(t) \approx 2\arcsin\sqrt{t}$ as $M$ grows and $\varepsilon\to0$, with variance normalization controlling the amplification of updates. The authors provide a practical recipe to approximate any smooth monotone $h$ by choosing coefficients $a_s,b_s$ so that $h_M'$ matches $h'$ in the Bernstein basis, with convergence guarantees as $M$ increases. The work emphasizes that RL fine-tuning methods are largely different monotone rescalings of the same underlying objective, suggesting that the best choice of $h$ depends on task context and optimization dynamics rather than finding a universally superior algorithm.

Abstract

We show that several popular algorithms for reinforcement learning in large language models with binary rewards can be viewed as stochastic gradient ascent on a monotone transform of the probability of a correct answer given a prompt. In particular, the transformation associated with rejection sampling algorithms is the logarithm and that associated with the GRPO algorithm is the arcsine of the square root.

Figures (3)

• Figure 1: GRPO loss vs. REINFORCE loss vs. log loss
• Figure 2: Function $h_M$ induced from rejection sampling.
• Figure 3: The effect of $M$ and $\varepsilon$ on the function $h_{M, \varepsilon}/h_{M, \varepsilon}(1)$.