Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning

Xichen Zhang, Sitong Wu, Yinghao Zhu, Haoru Tan, Shaozuo Yu, Ziyi He, Jiaya Jia

TL;DR

This work tackles the learning cliff in reinforcement learning from verifier rewards (RLVR) for LLM reasoning by introducing Scaf-GRPO, a scaffolded, on-policy training framework. It diagnoses stagnation through a guidance exemption phase and then employs a hierarchical hint hierarchy (knowledge, planning, solution) to inject minimal, progressively concrete prompts that enable the model to construct valid solutions while maintaining the GRPO objective. Empirical results across diverse math benchmarks and models show substantial gains over vanilla GRPO and prefix-based guidance (e.g., up to a 44.3% relative improvement on AIME24 for Qwen2.5-Math-7B) and demonstrated generalization to non-Qwen architectures and LongCoT settings, as well as strong OOD performance on GPQA-Diamond. The approach underscores the value of structured, in-prompt scaffolding for unlocking autonomous reasoning and provides a robust, model-agnostic pathway toward extending LLM capabilities while preserving exploration. Future work includes automating hint generation and adapting scaffolding to individual model proficiency levels, broadening applicability beyond strictly verifiable domains.

Abstract

Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the ''learning cliff'' phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24 benchmark by a relative 44.3% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous reasoning in LLM.

Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning

TL;DR

This work tackles the learning cliff in reinforcement learning from verifier rewards (RLVR) for LLM reasoning by introducing Scaf-GRPO, a scaffolded, on-policy training framework. It diagnoses stagnation through a guidance exemption phase and then employs a hierarchical hint hierarchy (knowledge, planning, solution) to inject minimal, progressively concrete prompts that enable the model to construct valid solutions while maintaining the GRPO objective. Empirical results across diverse math benchmarks and models show substantial gains over vanilla GRPO and prefix-based guidance (e.g., up to a 44.3% relative improvement on AIME24 for Qwen2.5-Math-7B) and demonstrated generalization to non-Qwen architectures and LongCoT settings, as well as strong OOD performance on GPQA-Diamond. The approach underscores the value of structured, in-prompt scaffolding for unlocking autonomous reasoning and provides a robust, model-agnostic pathway toward extending LLM capabilities while preserving exploration. Future work includes automating hint generation and adapting scaffolding to individual model proficiency levels, broadening applicability beyond strictly verifiable domains.

Abstract

Reinforcement learning from verifiable rewards has emerged as a powerful technique for enhancing the complex reasoning abilities of Large Language Models (LLMs). However, these methods are fundamentally constrained by the ''learning cliff'' phenomenon: when faced with problems far beyond their current capabilities, models consistently fail, yielding a persistent zero-reward signal. In policy optimization algorithms like GRPO, this collapses the advantage calculation to zero, rendering these difficult problems invisible to the learning gradient and stalling progress. To overcome this, we introduce Scaf-GRPO (Scaffolded Group Relative Policy Optimization), a progressive training framework that strategically provides minimal guidance only when a model's independent learning has plateaued. The framework first diagnoses learning stagnation and then intervenes by injecting tiered in-prompt hints, ranging from abstract concepts to concrete steps, enabling the model to construct a valid solution by itself. Extensive experiments on challenging mathematics benchmarks demonstrate Scaf-GRPO's effectiveness, boosting the pass@1 score of the Qwen2.5-Math-7B model on the AIME24 benchmark by a relative 44.3% over a vanilla GRPO baseline. This result demonstrates our framework provides a robust and effective methodology for unlocking a model's ability to solve problems previously beyond its reach, a critical step towards extending the frontier of autonomous reasoning in LLM.
Paper Structure (63 sections, 11 equations, 6 figures, 7 tables)

This paper contains 63 sections, 11 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement learning from verifier reward.
  4. RLVR with off-policy guidance.
  5. Methodology
  6. Preliminaries: Group Relative Policy Optimization (GRPO)
  7. The Scaf-GRPO Framework
  8. Phase 1: Diagnosing true-hard problems.
  9. Phase 2: Hierarchical hint-guided exploration.
  10. On-policy batch augmentation and unified loss.
  11. Conservative nature and on-policy integrity.
  12. Experiments
  13. Experimental Setups
  14. Training dataset.
  15. Models.
  16. ...and 48 more sections

Figures (6)

  • Figure 1: Scaf-GRPO overcomes the learning cliff with minimal guidance, outperforming vanilla GRPO grpo and the prefix-based LUFFY yan2025learning across challenging math benchmarks on Qwen2.5-Math-7B. By injecting strategic, hierarchical hints, our method unlocks the model's potential on difficult problems, achieving superior overall performance.
  • Figure 2: Training dynamics of Qwen2.5-Math-1.5B. (a) Scaf-GRPO overcomes the learning cliff by continuously solving zero-reward problems where vanilla GRPO plateaus. (b) This translates to sustained and superior validation accuracy for Scaf-GRPO throughout training.
  • Figure 3: Overview of the Scaf-GRPO framework. For a given query, the model generates multiple solutions. (Left) If any solution is correct, standard GRPO proceeds. (Right) If all solutions fail (the learning cliff), Scaf-GRPO initiates hierarchical hint-guided exploration. It injects progressively concrete in-prompt hints until a correct solution is found. This successful, minimally-guided trajectory replaces a failed one, restoring the learning gradient and enabling on-policy updates to resume.
  • Figure 4: Prompt for hint injection.
  • Figure 5: Evolution of reasoning from guidance to autonomy. The model progresses from imitating a concrete hint (a) to applying abstract knowledge (b), ultimately achieving (c) autonomous problem-solving by internalizing the guided skills.
  • ...and 1 more figures