Self-Hinting Language Models Enhance Reinforcement Learning

TL;DR

This work tackles the GRPO stall that arises under sparse terminal rewards by introducing SAGE, an on-policy reinforcement learning framework that injects privileged hints during training without changing the reward function. Hints are sampled with a tunable strength $\ell$ and appended to the input, with online self-hinting and a policy-dependent scheduler that activates hints only when within-group learning signals collapse, forming an automatic curriculum. The authors analyze GRPO as a gated update driven by the gate probability $u(p)=1-(1-p)^G-p^G$ and the energy $E=s^2/(s+\epsilon)^2$, showing that calibrating $p_\theta(x,h)$ toward $1/2$ yields more non-degenerate updates and better learning. Empirically, SAGE outperforms GRPO and other baselines across six benchmarks and three LLMs, with improved prompt utilization, faster learning on hard prompts, and robust generalization; online self-hinting and policy-dependent scheduling are key contributors. Overall, SAGE provides a scalable, on-policy mechanism to mitigate sparse-reward challenges in verifiable RL for LLMs, enabling more reliable alignment and reasoning capabilities.

Abstract

Group Relative Policy Optimization (GRPO) has recently emerged as a practical recipe for aligning large language models with verifiable objectives. However, under sparse terminal rewards, GRPO often stalls because rollouts within a group frequently receive identical rewards, causing relative advantages to collapse and updates to vanish. We propose self-hint aligned GRPO with privileged supervision (SAGE), an on-policy reinforcement learning framework that injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward. For each prompt $x$, the model samples a compact hint $h$ (e.g., a plan or decomposition) and then generates a solution $τ$ conditioned on $(x,h)$. Crucially, the task reward $R(x,τ)$ is unchanged; hints only increase within-group outcome diversity under finite sampling, preventing GRPO advantages from collapsing under sparse rewards. At test time, we set $h=\varnothing$ and deploy the no-hint policy without any privileged information. Moreover, sampling diverse self-hints serves as an adaptive curriculum that tracks the learner's bottlenecks more effectively than fixed hints from an initial policy or a stronger external model. Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO, on average +2.0 on Llama-3.2-3B-Instruct, +1.2 on Qwen2.5-7B-Instruct and +1.3 on Qwen3-4B-Instruct. The code is available at https://github.com/BaohaoLiao/SAGE.

Figures (7)

• Figure 1: An overview of our proposed method, SAGE. When an LLM can't sample any correct trajectory for a prompt, the LLM self-generates hint from the reference solution of the prompt. The hint is then used together with the difficult prompt as input to the LLM, avoiding advantage collapse and ensuring the sampling of correct trajectories to update the policy model.
• Figure 2: The percentage of prompts whose correct trajectories have NEVER been sampled w.r.t. the training step. Here we train on 64k prompts, and sample 8 traces per prompt per step. A large number of prompts is wasted during RL, especially for a weaker LLM, since they don't offer any signal for training.
• Figure 3: Average accuracy on Qwen3-4B-Instruct over 6 benchmarks. The 4.5k training prompts here are extremely hard, whose correct trajectories have never been sampled during training as Figure \ref{['fig:pass_k_0_during_training']}. The number of rollouts per prompt per step here is set to 32 to encourage exploration. Left: Performance on various hints. Training without hint only slightly improves the performance, since the reward signal from the hard prompts is sparse. However, training with any hint boosts the performance. Among all methods, online self-hinting consistently achieves the best performance across different hint levels. Right: Average accuracy w.r.t. the training steps for hint level $l=2$. Training without any hint even degrades the performance as the training goes, since the reward signal is too sparse, making it overfit to a few solvable prompts. However, online self-hinting boosts the performance steadily. Refer to Table \ref{['tab:detailed_number_for_hint_levels']} for detailed number.
• Figure 4: The training dynamics of different methods. For the training rewards, one should focus on the trend instead of the value, since adding hint (SAGE and Scaf-GRPO) modifies the prompt difficulty, and using a correct off-policy trajectory (LUFFY) increases the reward. (1) LUFFY shows the most instability, with a very high entropy for Llama and a very low reward at the beginning of training for Qwen3, because it imitates the off-policy trajectory whose distribution might not be aligned with the policy model. (2) Scaf-GRPO shows the lowest entropy, implying less exploration. (3) SAGE retains the on-policy characteristic, has a mild entropy and shows a stable growth in response length, which normally implies a better reasoning pattern.
• Figure 5: Ablation studies on Qwen3-4B-Instruct trained with the same prompt set as Figure \ref{['fig:poc_hints']}.

Theorems & Definitions (14)

• Corollary 3.1: Signal energy equals a gate probability
• Proposition 3.2: Gate opening probability under Bernoulli rewards
• Proposition 3.3: Optimal hint distribution is policy-dependent
• Remark 3.4: Why not sample many hints per prompt.
• proof
• proof
• proof
• proof
• Proposition 4.1: Expected non-standardized energy under Bernoulli rewards
• proof