Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning

TL;DR

RuscaRL tackles the exploration bottleneck in reinforcement learning for open-ended LLM reasoning by introducing rubric-based scaffolded exploration and verifiable rubric rewards. It combines explicit rubrics during rollout with a decaying scaffolding mechanism and a rubric-derived scalar reward to guide learning, implemented via GRPO. Across medical, writing, instruction-following, and STEM benchmarks, RuscaRL yields consistent gains, with small LLMs approaching or surpassing larger baselines and with strong improvements on weaker models. The approach offers a principled pathway to expand LLM reasoning boundaries and demonstrates robust, domain-general performance improvements, with code released for reproducing results.

Abstract

Recent advances in Large Language Models (LLMs) have underscored the potential of Reinforcement Learning (RL) to facilitate the emergence of reasoning capabilities. Despite the encouraging results, a fundamental dilemma persists as RL improvement relies on learning from high-quality samples, yet the exploration for such samples remains bounded by the inherent limitations of LLMs. This, in effect, creates an undesirable cycle in which what cannot be explored cannot be learned. In this work, we propose Rubric-Scaffolded Reinforcement Learning (RuscaRL), a novel instructional scaffolding framework designed to break the exploration bottleneck for general LLM reasoning. Specifically, RuscaRL introduces checklist-style rubrics as (1) explicit scaffolding for exploration during rollout generation, where different rubrics are provided as external guidance within task instructions to steer diverse high-quality responses. This guidance is gradually decayed over time, encouraging the model to internalize the underlying reasoning patterns; (2) verifiable rewards for exploitation during model training, where we can obtain robust LLM-as-a-Judge scores using rubrics as references, enabling effective RL on general reasoning tasks. Extensive experiments demonstrate the superiority of the proposed RuscaRL across various benchmarks, effectively expanding reasoning boundaries under the Best-of-N evaluation. Notably, RuscaRL significantly boosts Qwen2.5-7B-Instruct from 23.6 to 50.3 on HealthBench-500, surpassing GPT-4.1. Furthermore, our fine-tuned variant on Qwen3-30B-A3B-Instruct achieves 61.1 on HealthBench-500, outperforming leading LLMs including OpenAI-o3. Our code is available at https://github.com/IANNXANG/RuscaRL.

Figures (8)

• Figure 1: (Left) A conceptual illustration of exploration bottleneck and scaffolding. (Right) Performance comparison of different LLMs on HealthBench-500. Note that Qwen3-30B-Instruct denotes Qwen3-30B-A3B-Instruct, and Qwen3-235B-Thinking denotes Qwen3-235B-A22B-Thinking.
• Figure 2: Overview of the RuscaRL framework: (a) Rubric data sample with criteria and points, (b) Rubric-scaffolded reinforcement learning with reward computation and policy updates, (c) Intra-group scaffolding differentiation and inter-step scaffolding decay mechanisms.
• Figure 3: STEM Benchmarks.
• Figure 4: Best-of-N Performance.
• Figure 5: Training dynamics. The figure shows the evolution of policy entropy, validation accuracy, and sigmoid decay function during training.