Reinforcement Learning for Self-Improving Agent with Skill Library

TL;DR

The paper introduces SAGE, a reinforcement learning framework that endows self-improving agents with a reusable skill library. By employing Sequential Rollout across chains of similar tasks and a Skill-integrated Reward, SAGE encourages both the generation of high-quality executable skills and their effective reuse in subsequent tasks. Evaluated on AppWorld with a supervised-fine-tuned expert data regime, SAGE substantially improves Scenario Goal Completion and Task Goal Completion while reducing interaction steps and token usage, outperforming prior RL-based baselines. The work demonstrates the value of integrating skill libraries with RL for continual, efficient adaptation in unseen environments and provides extensive ablation analyses on retrieval, rewards, and initialization strategies.

Abstract

Large Language Model (LLM)-based agents have demonstrated remarkable capabilities in complex reasoning and multi-turn interactions but struggle to continuously improve and adapt when deployed in new environments. One promising approach is implementing skill libraries that allow agents to learn, validate, and apply new skills. However, current skill library approaches rely primarily on LLM prompting, making consistent skill library implementation challenging. To overcome these challenges, we propose a Reinforcement Learning (RL)-based approach to enhance agents' self-improvement capabilities with a skill library. Specifically, we introduce Skill Augmented GRPO for self-Evolution (SAGE), a novel RL framework that systematically incorporates skills into learning. The framework's key component, Sequential Rollout, iteratively deploys agents across a chain of similar tasks for each rollout. As agents navigate through the task chain, skills generated from previous tasks accumulate in the library and become available for subsequent tasks. Additionally, the framework enhances skill generation and utilization through a Skill-integrated Reward that complements the original outcome-based rewards. Experimental results on AppWorld demonstrate that SAGE, when applied to supervised-finetuned model with expert experience, achieves 8.9% higher Scenario Goal Completion while requiring 26% fewer interaction steps and generating 59% fewer tokens, substantially outperforming existing approaches in both accuracy and efficiency.

Figures (13)

• Figure 1: Illustration for Skill Library Agent and Sequential Rollout with Skill-integrated Reward.
• Figure 2: Analysis of Skill Usage Patterns. Performance metrics are shown as ratios relative to Base Model baseline, with numerical values annotated.
• Figure 3: The baseline agent directly generates codes to process the task.
• Figure 4: Skill Library Agent will first define a function and then call it to process the task.
• Figure 5: Training curve of SAGE.