Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks

Vishnu Sarukkai, Zhiqiang Xie, Kayvon Fatahalian

TL;DR

This work presents Traj-Bootstrap, a data-centric approach enabling LLM agents to autonomously improve through experience by constructing and curating a database of self-generated successful trajectories used as in-context exemplars. By combining trajectory collection with database-level and exemplar-level curation, the method yields substantial performance gains across ALFWorld, Wordcraft, and InterCode-SQL, surpassing several baselines and approaching or exceeding results from task-specific or hierarchical strategies. The study also analyzes training-time vs test-time costs, predicts agent success from the self-collected data, and demonstrates potential for fine-tuning with the same data. Overall, trajectory bootstrapping represents a scalable, less labor-intensive alternative to hand-crafted prompts and task-specific engineering, highlighting the importance of high-quality, reusable demonstration data in guiding LLM agents through complex, multi-step tasks.

Abstract

Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering--custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73% to 89%), Wordcraft (55% to 64%), and InterCode-SQL (75% to 79%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93% success on ALFWorld--surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.

Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks

TL;DR

This work presents Traj-Bootstrap, a data-centric approach enabling LLM agents to autonomously improve through experience by constructing and curating a database of self-generated successful trajectories used as in-context exemplars. By combining trajectory collection with database-level and exemplar-level curation, the method yields substantial performance gains across ALFWorld, Wordcraft, and InterCode-SQL, surpassing several baselines and approaching or exceeding results from task-specific or hierarchical strategies. The study also analyzes training-time vs test-time costs, predicts agent success from the self-collected data, and demonstrates potential for fine-tuning with the same data. Overall, trajectory bootstrapping represents a scalable, less labor-intensive alternative to hand-crafted prompts and task-specific engineering, highlighting the importance of high-quality, reusable demonstration data in guiding LLM agents through complex, multi-step tasks.

Abstract

Improving Large Language Model (LLM) agents for sequential decision-making tasks typically requires extensive task-specific knowledge engineering--custom prompts, curated examples, and specialized observation/action spaces. We investigate a different approach where agents automatically improve by learning from their own successful experiences without human intervention. Our method constructs and refines a database of self-generated trajectories that serve as in-context examples for future tasks. Even naive accumulation of successful trajectories yields substantial performance gains across three diverse benchmarks: ALFWorld (73% to 89%), Wordcraft (55% to 64%), and InterCode-SQL (75% to 79%). These improvements exceed those achieved by upgrading from gpt-4o-mini to gpt-4o and match the performance of allowing multiple attempts per task. We further enhance this approach with two innovations: database-level curation using population-based training to propagate high-performing example collections, and exemplar-level curation that selectively retains trajectories based on their empirical utility as in-context examples. With these enhancements, our method achieves 93% success on ALFWorld--surpassing approaches that use more powerful LLMs and hand-crafted components. Our trajectory bootstrapping technique demonstrates that agents can autonomously improve through experience, offering a scalable alternative to labor-intensive knowledge engineering.
Paper Structure (60 sections, 4 equations, 7 figures, 5 tables, 4 algorithms)

This paper contains 60 sections, 4 equations, 7 figures, 5 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Sequential Decision-Making Tasks
  4. ReAct-style Agent Loop
  5. Problem Statement
  6. Related Work
  7. In-context learning for agent improvement
  8. Automatic in-context examples
  9. In-context self-improvement of LLM Agents
  10. Methods
  11. Traj-Bootstrap: Constructing a Database of Previously-Solved Tasks
  12. +DB-Curation: Database-Level Data Curation
  13. +Exemplar-Curation: Exemplar-Level Data Curation
  14. Train-Time vs Test-Time LLM Costs
  15. Experiments
  16. ...and 45 more sections

Figures (7)

  • Figure 1: Traj-Bootstrap leads to variance in test-time success rate. Individual trials (5) shown as dashed lines, results on Intercode-SQL benchmark. There is noticeable variability in performance across trials.
  • Figure 2: Traj-Bootstrap results: success rate improves with increasing training tasks on all three benchmarks. Individual trials (5) shown as dashed lines. All benchmarks exhibit diminishing returns as the database size increases. Trials show substantial performance variability, both within individual trials and across different trials.
  • Figure 3: Success rate comparison for Traj-Bootstrap and its variants (+DB-Curation, +Exemplar-Curation, +DB+Exemplar-Curation). +DB-Curation enhances final success rate only on ALFWorld, but improves success rate for smaller DB sizes on all benchmarks. +Exemplar-Curation delivers success rate gains on both Intercode-SQL and Wordcraft. The combination of both enhancements delivers the largest gains on both ALFWorld and InterCode-SQL.
  • Figure 4: The 'best' bootstrapped trajectories compared to the 'worst'. Databases constructed from the highest-quality successful trajectory per task, as measured by Eq. \ref{['eq:exemplar_metric']}, outperform databases built from the lowest-quality successful trajectories on both ALFWorld and Wordcraft. The 'best' curve is identical to +Exemplar-Curation, while the 'worst' curve selects the bottom-1 trajectory instead of top-1 in Alg. \ref{['alg:exemplar_curation']}, line 7.
  • Figure 5: AUROC of success prediction improves with more self-collected examples. Performance continues to rise with increasing database size.
  • ...and 2 more figures