Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoRefine: From Trajectories to Reusable Expertise for Continual LLM Agent Refinement

Libin Qiu, Zhirong Gao, Junfu Chen, Yuhang Ye, Weizhi Huang, Xiaobo Xue, Wenkai Qiu, Shuo Tang

TL;DR

AutoRefine tackles the problem of continual knowledge accumulation for LLM-based agents by introducing dual-form Experience Patterns: Skill Patterns for static procedural knowledge and Subagent Patterns for complex, stateful subtasks. A continuous maintenance mechanism scores, prunes, and merges patterns to prevent repository degradation, enabling scalable and reusable knowledge over time. Empirical results across ALFWorld, ScienceWorld, and TravelPlanner show strong performance gains, including automatic extraction surpassing some manual baselines and substantial reductions in task steps. The framework demonstrates that combining pattern-based procedural guidance with autonomous subagents yields robust cross-task transfer and improved planning under constraints, with practical implications for deploying more reliable and scalable autonomous agents. Future work points to incorporating failure-driven learning, adaptive hyperparameters, and broader domain transfer to further enhance the approach’s applicability.

Abstract

Large language model agents often fail to accumulate knowledge from experience, treating each task as an independent challenge. Recent methods extract experience as flattened textual knowledge, which cannot capture procedural logic of complex subtasks. They also lack maintenance mechanisms, causing repository degradation as experience accumulates. We introduce AutoRefine, a framework that extracts and maintains dual-form Experience Patterns from agent execution histories. For procedural subtasks, we extract specialized subagents with independent reasoning and memory. For static knowledge, we extract skill patterns as guidelines or code snippets. A continuous maintenance mechanism scores, prunes, and merges patterns to prevent repository degradation. Evaluated on ALFWorld, ScienceWorld, and TravelPlanner, AutoRefine achieves 98.4%, 70.4%, and 27.1% respectively, with 20-73% step reductions. On TravelPlanner, automatic extraction exceeds manually designed systems (27.1% vs 12.1%), demonstrating its ability to capture procedural coordination.

AutoRefine: From Trajectories to Reusable Expertise for Continual LLM Agent Refinement

TL;DR

AutoRefine tackles the problem of continual knowledge accumulation for LLM-based agents by introducing dual-form Experience Patterns: Skill Patterns for static procedural knowledge and Subagent Patterns for complex, stateful subtasks. A continuous maintenance mechanism scores, prunes, and merges patterns to prevent repository degradation, enabling scalable and reusable knowledge over time. Empirical results across ALFWorld, ScienceWorld, and TravelPlanner show strong performance gains, including automatic extraction surpassing some manual baselines and substantial reductions in task steps. The framework demonstrates that combining pattern-based procedural guidance with autonomous subagents yields robust cross-task transfer and improved planning under constraints, with practical implications for deploying more reliable and scalable autonomous agents. Future work points to incorporating failure-driven learning, adaptive hyperparameters, and broader domain transfer to further enhance the approach’s applicability.

Abstract

Large language model agents often fail to accumulate knowledge from experience, treating each task as an independent challenge. Recent methods extract experience as flattened textual knowledge, which cannot capture procedural logic of complex subtasks. They also lack maintenance mechanisms, causing repository degradation as experience accumulates. We introduce AutoRefine, a framework that extracts and maintains dual-form Experience Patterns from agent execution histories. For procedural subtasks, we extract specialized subagents with independent reasoning and memory. For static knowledge, we extract skill patterns as guidelines or code snippets. A continuous maintenance mechanism scores, prunes, and merges patterns to prevent repository degradation. Evaluated on ALFWorld, ScienceWorld, and TravelPlanner, AutoRefine achieves 98.4%, 70.4%, and 27.1% respectively, with 20-73% step reductions. On TravelPlanner, automatic extraction exceeds manually designed systems (27.1% vs 12.1%), demonstrating its ability to capture procedural coordination.
Paper Structure (72 sections, 7 equations, 6 figures, 8 tables)

This paper contains 72 sections, 7 equations, 6 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning Paradigms for Agent Improvement
  4. Training-Based Methods.
  5. In-Context Learning with External Data.
  6. In-Context Learning with Self-Generated Experience.
  7. Knowledge Extraction from Agent Experience
  8. Self-Reflection and Feedback.
  9. Experience as Textual Knowledge.
  10. Methodology
  11. Problem Formulation
  12. Pattern Repository.
  13. Pattern Metadata.
  14. Framework Overview
  15. Task Execution with Patterns
  16. ...and 57 more sections

Figures (6)

  • Figure 1: Overview of AutoRefine framework with three stages: task execution, pattern extraction, and pattern maintenance.
  • Figure 2: Ablation study results on TravelPlanner validation set. Subagent removal causes larger drops on commonsense constraints (57% relative decrease) than hard constraints (24% relative decrease).
  • Figure 3: Results of (a) Repository size and (b) Pattern utilization rate ($u_j/r_j$) over training.
  • Figure 4: Hyperparameter sensitivity for batch size $K$, pruning percentile $\alpha$, and retrieval count $k$. (a) ScienceWorld peaks at $K=15$ while others peak at $K=10$. (b) Optimal range $\alpha \in [15\%, 25\%]$ across all benchmarks. (c) TravelPlanner shows 24% drop at low $k=5$. Green shading indicates optimal ranges.
  • Figure 5: Similarity threshold $\theta$ analysis. (a) TravelPlanner peaks at $\theta=0.6$ (36.2%) rather than default $\theta=0.5$ (35.6%), while ALFWorld and ScienceWorld peak at 0.5. (b) Breakdown shows higher $\theta$ improves hard constraint satisfaction (41.3% vs. 38.9%).
  • ...and 1 more figures