Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration

Yang Zhao, Yangou Ouyang, Xiao Ding, Hepeng Wang, Bibo Cai, Kai Xiong, Jinglong Gao, Zhouhao Sun, Li Du, Bing Qin, Ting Liu

TL;DR

PRISM addresses the data allocation bottleneck between supervised fine-tuning and reinforcement learning in LLM agents by diagnosing cognitive conflict through gradient concentration. It introduces a three-stage pipeline—non-invasive gradient probing, structural dissonance quantification via Gini, Kurtosis, and CV, and distribution-adaptive routing using a median split—to route low-conflict data to SFT and high-conflict data to RL. Empirically, PRISM yields Pareto improvements on WebShop and ALFWorld, achieving state-of-the-art Seen performance on ALFWorld (95.31%) and up to 3.22× training speedups with only half the RL budget, while remaining backbone-agnostic across Qwen and Llama. These results support the central claim that disentangling data by internal optimization regimes enhances robustness and efficiency in scalable agent alignment.

Abstract

While Hybrid Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become the standard paradigm for training LLM agents, effective mechanisms for data allocation between these stages remain largely underexplored. Current data arbitration strategies often rely on surface-level heuristics that fail to diagnose intrinsic learning needs. Since SFT targets pattern consolidation through imitation while RL drives structural adaptation via exploration, misaligning data with these functional roles causes severe optimization interference. We propose PRISM, a dynamics-aware framework grounded in Schema Theory that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge. By analyzing the spatial geometric structure of gradients, PRISM identifies data triggering high spatial concentration as high-conflict signals that require RL for structural restructuring. In contrast, data yielding diffuse updates is routed to SFT for efficient consolidation. Extensive experiments on WebShop and ALFWorld demonstrate that PRISM achieves a Pareto improvement, outperforming state-of-the-art hybrid methods while reducing computational costs by up to 3.22$\times$. Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.

Consolidation or Adaptation? PRISM: Disentangling SFT and RL Data via Gradient Concentration

TL;DR

PRISM addresses the data allocation bottleneck between supervised fine-tuning and reinforcement learning in LLM agents by diagnosing cognitive conflict through gradient concentration. It introduces a three-stage pipeline—non-invasive gradient probing, structural dissonance quantification via Gini, Kurtosis, and CV, and distribution-adaptive routing using a median split—to route low-conflict data to SFT and high-conflict data to RL. Empirically, PRISM yields Pareto improvements on WebShop and ALFWorld, achieving state-of-the-art Seen performance on ALFWorld (95.31%) and up to 3.22× training speedups with only half the RL budget, while remaining backbone-agnostic across Qwen and Llama. These results support the central claim that disentangling data by internal optimization regimes enhances robustness and efficiency in scalable agent alignment.

Abstract

While Hybrid Supervised Fine-Tuning (SFT) followed by Reinforcement Learning (RL) has become the standard paradigm for training LLM agents, effective mechanisms for data allocation between these stages remain largely underexplored. Current data arbitration strategies often rely on surface-level heuristics that fail to diagnose intrinsic learning needs. Since SFT targets pattern consolidation through imitation while RL drives structural adaptation via exploration, misaligning data with these functional roles causes severe optimization interference. We propose PRISM, a dynamics-aware framework grounded in Schema Theory that arbitrates data based on its degree of cognitive conflict with the model's existing knowledge. By analyzing the spatial geometric structure of gradients, PRISM identifies data triggering high spatial concentration as high-conflict signals that require RL for structural restructuring. In contrast, data yielding diffuse updates is routed to SFT for efficient consolidation. Extensive experiments on WebShop and ALFWorld demonstrate that PRISM achieves a Pareto improvement, outperforming state-of-the-art hybrid methods while reducing computational costs by up to 3.22. Our findings suggest that disentangling data based on internal optimization regimes is crucial for scalable and robust agent alignment.
Paper Structure (47 sections, 6 equations, 5 figures, 4 tables)

This paper contains 47 sections, 6 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Gradient Concentration as a Proxy for Cognitive Conflict
  4. Stage I: Non-Invasive Gradient Probing
  5. Stage II: Quantifying Structural Dissonance
  6. Stage III: Distribution-Adaptive Routing
  7. Experiments
  8. Experimental Setup
  9. Benchmarks.
  10. Implementation Details.
  11. Baselines
  12. Main Results
  13. Generalization and Efficiency on ALFWorld
  14. Backbone-Agnostic Robustness on WebShop
  15. Ablation and Analysis
  16. ...and 32 more sections

Figures (5)

  • Figure 1: Case Study on ALFWorld. PRISM performs data arbitration by diagnosing cognitive conflict between task trajectories and the model's internal state. (Left) Case A: A routine task follows a linear execution sequence, characterized by diffuse gradient updates (low concentration). Such samples are routed to SFT for behavioral consolidation. (Right) Case B: A high-conflict task involving extensive trial-and-error (e.g., searching multiple locations) triggers concentrated gradient updates (high concentration). These signals indicate a failure in the model's current logic, necessitating RL for structural adaptation and reasoning refinement.
  • Figure 2: Overview of PRISM. The framework consists of three stages: (1) Non-Invasive Gradient Probing: Extracting update landscapes to capture internal reactions to each sample; (2) Quantifying Structural Dissonance: Measuring gradient concentration to diagnose the conflict between the data and existing knowledge; (3) Distribution-Adaptive Routing: Partitioning data based on concentration—samples with low-conflict (diffuse updates) are routed to SFT for consolidation, while those with high conflict (concentrated updates) are routed to RL for structural restructuring.
  • Figure 3: Ablation study of data routing strategies on WebShop. We compare PRISM (orange) with the Inverse allocation (blue: SFT on high-conflict data and RL on low-conflict data) under three concentration metrics for (a) Qwen3-8B and (b) Llama-3.1-8B-Instruct. The dashed line denotes the Random Baseline.
  • Figure 4: Sensitivity to RL Allocation Ratio. Performance of Qwen3-8B on WebShop (CV metric) across varying RL data proportions. The observed inverted U-shape peaks at a 50% split, indicating that a balanced allocation yields optimal performance compared to insufficient adaptation or excessive exploration.
  • Figure 5: Venn Diagram of Data Selection Consensus. The intersection shows that approximately 60% of the high-conflict trajectories are consistently identified by all three statistical metrics. This high degree of convergence significantly exceeds the 12.5%--25.0% expected from random overlapping splits, demonstrating that PRISM captures a stable underlying structural dissonance signal regardless of the specific concentration metric employed.