Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting

TL;DR

The paper tackles the challenge of sample-efficient online learning for language-model agents operating in novel environments. It introduces ECHO, a prompting framework that adapts hindsight experience replay to LM agents, enabling the generation and learning from counterfactual trajectories through a hindsight rule and a memory-update rule. Empirical evaluation on stateful variants of XMiniGrid and PeopleJoinQA shows that ECHO substantially outperforms vanilla baselines (up to 80% in mean reward on XMiniGrid-Stateful) and often surpasses or matches more sophisticated baselines in efficiency and accuracy. The work demonstrates that LM agents can leverage their own world knowledge to rewrite and compress past experiences, yielding faster adaptation in partially observable settings, and it provides two new benchmarks for evaluating such learning dynamics.

Abstract

Language model (LM) agents deployed in novel environments often exhibit poor sample efficiency when learning from sequential interactions. This significantly hinders the usefulness of such agents in environments where interaction is costly (for example, when they interact with humans or reset physical systems). While a number of existing LM agent architectures incorporate various mechanisms for experience storage and reflection, they make limited use of LMs' abilities to directly generate or reason about full counterfactual trajectories. We introduce ECHO (Experience Consolidation via Hindsight Optimization), a prompting framework that adapts hindsight experience replay from reinforcement learning for language model agents. ECHO generates optimized trajectories for alternative goals that could have been achieved during failed attempts, effectively creating synthetic positive examples from unsuccessful interactions. Our approach consists of two components: a hindsight rule that uses the language model itself to identify relevant subgoals and generate optimized trajectories, and an update rule that maintains compressed trajectory representations in memory. We evaluate ECHO on stateful versions of XMiniGrid, a text-based navigation and planning benchmark, and PeopleJoinQA, a collaborative information-gathering enterprise simulation. Across both domains, ECHO outperforms vanilla language agent baselines by up to 80%; in XMiniGrid, it also outperforms a number of sophisticated agent architectures including Reflexion and AWM, demonstrating faster adaptation to novel environments through more effective utilization of past experiences.

Figures (4)

• Figure 1: ECHO in the XMiniGrid environment. The agent fails in its first trajectory (left). Using this trajectory, ECHO identifies other objects the agent could have reached, and generates an optimized trajectory for these goals (right). In future iterations, the agent can then use these optimized trajectories to successfully complete unseen goals.
• Figure 2: Results on the XMiniGrid-Stateful benchmark. Left: ECHO achieves the highest mean reward. Right: ECHO's cumulative reward is higher than the baseline ReACT agent's after 3 interactions, indicating that ECHO improves compared to a static baseline over time.
• Figure 3: Results on the PeopleJoinQA-Stateful benchmark. Left: While Reflexion achieves slightly higher accuracy, ECHO and AWM are more efficient, completing the task in 1.6 fewer messages on average. Right: we plot the running average reward gain above ReAct. On average, ECHO outperforms the ReAct after the first query.
• Figure 4: We chose 5 organizations from PeopleJoinQA, maximizing variation amongst number of people in the organization and total number of queries. No offline method consistently outperforms the baseline on both accuracy and efficiency for all organizations.