GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning

TL;DR

GHPO introduces a difficulty-aware reinforcement learning framework for RLVR that automatically detects problem difficulty and adaptively refines prompts to balance imitation learning and on-policy RL. By combining automated difficulty detection with multi-stage hinting, GHPO mitigates reward sparsity and stabilizes training while improving sample efficiency. Empirical results across six challenging mathematics benchmarks show about a 5% average improvement over strong baselines and robust generalization across model families. The approach enables scalable, data-efficient reasoning enhancements for on-device LLMs by tailoring guidance to the model's evolving capabilities.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has recently emerged as a powerful paradigm for facilitating the self-improvement of large language models (LLMs), particularly in the domain of complex reasoning tasks. However, prevailing on-policy RL methods often contend with significant training instability and inefficiency. This is primarily due to a capacity-difficulty mismatch, where the complexity of training data frequently outpaces the model's current capabilities, leading to critically sparse reward signals and stalled learning progress. This challenge is particularly acute for smaller, more resource-efficient LLMs. To overcome this, we introduce the Guided Hybrid Policy Optimization (GHPO), a novel difficulty-aware reinforcement learning framework. GHPO dynamically calibrates task difficulty by employing adaptive prompt refinement to provide targeted guidance. This unique approach adaptively balances direct imitation learning for problems currently beyond the model's reach with exploration-based reinforcement learning for more manageable tasks, effectively creating a smooth and optimized learning curriculum. Extensive experiments demonstrate that GHPO achieves an average performance gain of approximately 5% across six challenging mathematics benchmarks, consistently outperforming strong on-policy reinforcement learning and curriculum learning baselines. Further analysis confirms that our framework significantly enhances both training stability and final reasoning performance, thus offering a scalable and efficient solution for developing powerful and robust reasoning models.

Figures (6)

• Figure 1: Overall Performance of GHPO across several benchmarks on the Qwen2.5-7B base model.
• Figure 2: An illustration of the proposed GHPO framework.
• Figure 3: Prompt template used in the proposed GHPO framework.
• Figure 4: An illustration of using the proposed GHPO for addressing a detected difficult problem with 50% hint extraction.
• Figure 5: The proportion of problems detected to be difficult within a mini-batch.