HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness

TL;DR

The paper addresses reward sparsity in reinforcement learning for large language model reasoning by identifying low training affinity as a core failure mode when integrating off-policy data. It introduces Affinity, a diagnostic metric combining update effectiveness and stability via EUR and UC, to monitor training dynamics. Building on this, it proposes HINT, an adaptive two-stage rollout that provides heuristic hints from a teacher only when rewards are sparse, preserving the model's autonomous reasoning while avoiding answer leakage. Empirical results on mathematical reasoning benchmarks show that HINT achieves state-of-the-art performance across model scales, with improved data efficiency and robust out-of-distribution generalization, supported by analyses of training dynamics and exploration quality.

Abstract

Reinforcement Learning (RL) has become a key driver for enhancing the long chain-of-thought (CoT) reasoning capabilities of Large Language Models (LLMs). However, prevalent methods like GRPO often fail when task difficulty exceeds the model's capacity, leading to reward sparsity and inefficient training. While prior work attempts to mitigate this using off-policy data, such as mixing RL with Supervised Fine-Tuning (SFT) or using hints, they often misguide policy updates In this work, we identify a core issue underlying these failures, which we term low training affinity. This condition arises from a large distributional mismatch between external guidance and the model's policy. To diagnose this, we introduce Affinity, the first quantitative metric for monitoring exploration efficiency and training stability. To improve Affinity, we propose HINT: Helping Ineffective rollouts Navigate Towards effectiveness, an adaptive hinting framework. Instead of providing direct answers, HINT supplies heuristic hints that guide the model to discover solutions on its own, preserving its autonomous reasoning capabilities. Extensive experiments on mathematical reasoning tasks show that HINT consistently outperforms existing methods, achieving state-of-the-art results with models of various scales, while also demonstrating significantly more stable learning and greater data efficiency.Code is available on Github.

Figures (7)

• Figure 1: Comparison of Hint Mechanisms and Their Impact on Learning. The answer-level hint provides an explicit partial solution. The model can achieve a reward by simply completing this pre-defined path, which encourages learning a superficial shortcut rather than genuine reasoning. In contrast, our heuristic hint offers a high-level conceptual prompt, compelling the model to develop its own solution path independently.
• Figure 2: A comparison of training rewards (top) and test accuracy (bottom). High rewards during training do not necessarily lead to high test accuracy, indicating that reward signals may be misleading indicators of model generalization.
• Figure 3: The HINT Framework: An Adaptive Two-Stage Rollout Process. HINT operates in two stages. (1) Standard Rollout: The model first samples trajectories from the original problem. If the rewards are non-sparse (at least one is correct), the process follows the standard GRPO update path. (2) Hint-Augmented Rollout: If, however, the rewards are sparse (all trajectories are incorrect), the hint mechanism is activated. The model then re-rolls out conditioned on a heuristic hint from a "teacher model". This stage is designed to produce non-sparse rewards, turning a failed sample into a valuable learning opportunity.
• Figure 4: We record the EUR, UC, and Affinity metrics across different training processes to investigate the impact of various off-policy strategies on training. Left: EUR during training; Middle: UC during training; Right:Affinity during training. Overall, HINT most effectively alleviates the EUR collapse, avoids persistently high UC, and achieves higher Affinity, thereby enabling more stable and efficient training.
• Figure 5: Sampling Efficiency of HINT and GRPO at Different Training Stages. Under an equal budget, HINT yields 1,485 more valid samples (top) and achieves a 18.9% higher final proportion of valid samples (bottom).