Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Dynamic Dual-Granularity Skill Bank for Agentic RL

Songjun Tu, Chengdong Xu, Qichao Zhang, Yaocheng Zhang, Xiangyuan Lan, Linjing Li, Dongbin Zhao

Abstract

Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10-20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Dynamic Dual-Granularity Skill Bank for Agentic RL

Abstract

Agentic reinforcement learning (RL) can benefit substantially from reusable experience, yet existing skill-based methods mainly extract trajectory-level guidance and often lack principled mechanisms for maintaining an evolving skill memory. We propose D2Skill, a dynamic dual-granularity skill bank for agentic RL that organizes reusable experience into task skills for high-level guidance and step skills for fine-grained decision support and error correction. D2Skill jointly trains the policy and skill bank through paired baseline and skill-injected rollouts under the same policy, using their performance gap to derive hindsight utility signals for both skill updating and policy optimization. Built entirely from training-time experience, the skill bank is continuously expanded through reflection and maintained with utility-aware retrieval and pruning. Experiments on ALFWorld and WebShop with Qwen2.5-7B-Instruct and Qwen3-4B-Instruct-2507 show that D2Skill consistently improves success rates over skill-free baselines by 10-20 points. Further ablations and analyses show that both dual-granularity skill modeling and dynamic skill maintenance are critical to these gains, while the learned skills exhibit higher utility, transfer across evaluation settings, and introduce only modest training overhead.

Paper Structure

This paper contains 29 sections, 10 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Agentic RL as a History-Augmented Decision Process
  4. Skill Bank as an External Knowledge Store
  5. Method
  6. Overall Framework
  7. RL training with skill injection. (Section \ref{['sec_method_1']})
  8. Reflection-driven skill generation. (Section \ref{['sec_method_2']})
  9. Skill retrieval and bank management. (Section \ref{['sec_method_2']})
  10. RL Training with Skill Injection and Hindsight Optimization
  11. Rollout with skill injection.
  12. Hindsight signals and utility updates.
  13. Hindsight intrinsic reward shaping.
  14. Policy optimization with skill-augmented returns.
  15. Skill Generation, Retrieval, and Bank Management
  16. ...and 14 more sections

Figures (5)

  • Figure 1: Overview of D2Skill. (a) The dynamic dual-granularity skill bank with retrieval, reflection-driven generation, and management. (b) Overall results on ALFWorld and WebShop. (c) ALFWorld training curves for the D2Skill skill group, paired baseline group, and GRPO. (d) Skill bank dynamics with and without management, shown by average skill utility and retrieval statistics.
  • Figure 2: Overall framework of D2Skill. D2Skill couples RL with a dynamic dual-granularity skill bank. For each task, training rollouts are divided into a baseline group and a skill group, whose performance gap yields hindsight signals for policy optimization and skill utility estimation. When performance is poor, reflection on representative failed trajectories produces task skills for high-level guidance and step skills for local error correction. Skills are stored with retrieval keys, reused during subsequent interaction, and periodically pruned by utility-based bank management.
  • Figure 3: Eval with Different Skills.
  • Figure 3: Training Cost.
  • Figure 4: Val Success Dynamics.