Contextual Experience Replay for Self-Improvement of Language Agents

TL;DR

CER is a training-free framework that enables self-improvement of language agents by distilling environment dynamics and decision-making patterns from past trajectories into a dynamic memory buffer. Retrieved experiences are transformed into in-context, NL prompts and replayed within the agent's context to guide decisions across online, offline, and hybrid learning settings. Empirical results on WebArena and VisualWebArena show CER markedly improves baseline GPT-4o, demonstrates strong token efficiency, stability-plasticity trade-offs, and compatibility with other methods. The work highlights CER's potential to enable environment-specific adaptation in autonomous language agents with modest additional costs and broad applicability beyond the studied benchmarks.

Abstract

Large language model (LLM) agents have been applied to sequential decision-making tasks such as web navigation, but without any environment-specific experiences, they often fail in these complex tasks. Moreover, current LLM agents are not designed to continually learn from past experiences during inference time, which could be crucial for them to gain these environment-specific experiences. To address this, we propose Contextual Experience Replay (CER), a training-free framework to enable efficient self-improvement for language agents in their context window. Specifically, CER accumulates and synthesizes past experiences into a dynamic memory buffer. These experiences encompass environment dynamics and common decision-making patterns, allowing the agents to retrieve and augment themselves with relevant knowledge in new tasks, enhancing their adaptability in complex environments. We evaluate CER on the challenging WebArena and VisualWebArena benchmarks. On VisualWebArena, CER achieves a competitive performance of 31.9%. On WebArena, CER also gets a competitive average success rate of 36.7%, relatively improving the success rate of the GPT-4o agent baseline by 51.0%. We also conduct a comprehensive analysis on it to prove its efficiency, validity and understand it better.

Figures (9)

• Figure 1: Overview of Contextual Experience Replay including offline and online settings. (1) In the online setting, it will start from stage C and loop between stage C and B for each task, i.e. solve task $i$, learn experiences from it and solve task $i+1$ with previous experiences, and so on. (2) In the offline setting, stage A is needed to get offline trajectories, then it goes from stage B to C and finally stays in stage C, i.e., learns experiences from offline trajectories and solves all tasks. (3) In the hybrid setting, it will begin from stage A and loop between B and C, conducting both offline and online learning.
• Figure 2: Compare ReAct baseline with ReAct + CER. The experiences, including dynamics and skills, are obtained through multiple modules as in Fig.\ref{['fig:off_on']}. They are "replayed" in the context window of the model, helping the agent to make correct decisions. For simplicity, the thinking process is neglected in the figure.
• Figure 3: System message for dynamics distillation module in CER
• Figure 4: System message for skills distillation module in CER
• Figure 5: System message for dynamics retrieval module in CER