Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

Yuxuan Jiang, Francis Ferraro

TL;DR

SCRIBE introduces a mid-level supervision framework for tool-using LLMs by grounding reward signals in a curated library of Skill Prototypes and routing trajectories to structured <subgoal, skill, step> representations. A prototype-conditioned reward model, calibrated by multiple prompts, provides dense, diagnostics-style feedback that reduces reward variance and stabilizes training. Empirical results show state-of-the-art performance on mathematical reasoning (e.g., AIME25) and multi-turn tool-use benchmarks, with evidence of a co-evolution between mid-level skill mastery and emergent high-level planning. SCRIBE additionally proves additive with low-level tool optimizations, offering a scalable path toward more autonomous and reliable tool-using agents.

Abstract

Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.

SCRIBE: Structured Mid-Level Supervision for Tool-Using Language Models

TL;DR

SCRIBE introduces a mid-level supervision framework for tool-using LLMs by grounding reward signals in a curated library of Skill Prototypes and routing trajectories to structured <subgoal, skill, step> representations. A prototype-conditioned reward model, calibrated by multiple prompts, provides dense, diagnostics-style feedback that reduces reward variance and stabilizes training. Empirical results show state-of-the-art performance on mathematical reasoning (e.g., AIME25) and multi-turn tool-use benchmarks, with evidence of a co-evolution between mid-level skill mastery and emergent high-level planning. SCRIBE additionally proves additive with low-level tool optimizations, offering a scalable path toward more autonomous and reliable tool-using agents.

Abstract

Training reliable tool-augmented agents remains a significant challenge, largely due to the difficulty of credit assignment in multi-step reasoning. While process-level reward models offer a promising direction, existing LLM-based judges often produce noisy and inconsistent signals because they lack fine-grained, task-specific rubrics to distinguish high-level planning from low-level execution. In this work, we introduce SCRIBE (Skill-Conditioned Reward with Intermediate Behavioral Evaluation), a reinforcement learning framework that intervenes at a novel mid-level abstraction. SCRIBE grounds reward modeling in a curated library of skill prototypes, transforming open-ended LLM evaluation into a constrained verification problem. By routing each subgoal to a corresponding prototype, the reward model is equipped with precise, structured rubrics that substantially reduce reward variance. Experimental results show that SCRIBE achieves state-of-the-art performance across a range of reasoning and tool-use benchmarks. In particular, it improves the AIME25 accuracy of a Qwen3-4B model from 43.3% to 63.3%, and significantly increases success rates in complex multi-turn tool interactions. Further analysis of training dynamics reveals a co-evolution across abstraction levels, where mastery of mid-level skills consistently precedes the emergence of effective high-level planning behaviors. Finally, we demonstrate that SCRIBE is additive to low-level tool optimizations, providing a scalable and complementary pathway toward more autonomous and reliable tool-using agents.
Paper Structure (58 sections, 8 equations, 3 figures, 5 tables)

This paper contains 58 sections, 8 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Tool-Integrated Reasoning with Agentic Reinforcement Learning
  4. Progress Reward Modeling with LLM-as-Judge
  5. Problem Decomposition and skills in Reasoning
  6. Method: Decomposition, Clustering, prototype
  7. Mid-Level Skill Abstraction
  8. Prototype Construction.
  9. Structured Verification via Prototypes.
  10. Skill-Conditioned Reward and Optimization
  11. Routing and Reward Evaluation.
  12. Policy Optimization with Adaptive Prototypes.
  13. Experimental Setup
  14. Experimental Settings.
  15. Baselines.
  16. ...and 43 more sections

Figures (3)

  • Figure 1: Overview of our three-stage framework. The policy model performs high-level planning, mid-level reasoning, and low-level execution. At the mid level, a router retrieves skill-specific prototypes from a prototype library and uses them to structure reward evaluation by a frozen LLM-based (GPT5-mini) reward model. The policy is optimized with GRPO using the resulting rewards. Flame denotes trainable components, while snowflake indicates frozen models.
  • Figure 2: A compact illustration of a skill prototype used for cluster-calibrated judging. Detailed prototypes and trap-to-score mappings are deferred to the appendix.
  • Figure 3: Structural training dynamics of our method. From top to bottom, we report mid-level uncertainty, high-level plan selection ability, plan separability, and final task accuracy. Results are shown separately for math (left column) and tool-use (right column) on Qwen3-4B-SCRIBE.