RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback

TL;DR

RetroAgent is introduced, an online RL framework that enables agents to master complex interactive environments not only by solving, but also by evolving under the joint guidance of extrinsic task-success rewards and retrospective dual intrinsic feedback.

Abstract

Large language model (LLM)-based agents trained with reinforcement learning (RL) have shown strong potential on complex interactive tasks. However, standard RL paradigms favor static problem-solving over continuous adaptation: agents often converge to suboptimal strategies due to insufficient exploration, while learned knowledge remains implicit within parameters rather than explicitly retrievable, limiting effective experiential learning. To address these limitations, we introduce RetroAgent, an online RL framework that empowers agents to master complex interactive environments not just by solving, but by evolving. Concretely, RetroAgent features a hindsight self-reflection mechanism that produces dual intrinsic feedback: (1) intrinsic numerical feedback that that tracks incremental subtask completion relative to prior attempts, rewarding promising explorations, and (2) intrinsic language feedback that distills reusable lessons into a memory buffer, retrieved via our proposed Similarity & Utility-Aware Upper Confidence Bound (SimUtil-UCB) strategy balancing relevance, utility, and exploration to effectively leverage past experiences. Extensive experiments on two model families across four challenging agentic tasks demonstrate that RetroAgent significantly outperforms existing methods, achieving state-of-the-art results -- e.g., surpassing Group Relative Policy Optimization (GRPO)-trained agents by +18.3% on ALFWorld, +15.4% on WebShop, +27.1% on Sokoban, and +8.9% on MineSweeper -- while exhibiting strong test-time adaptation and generalization to out-of-distribution scenarios.

Figures (12)

• Figure 1: (a) Overview of the RetroAgent framework. After each episode, the agent analyzes its trajectory via a self-reflection mechanism to produce dual intrinsic feedback, enabling effective learning from past experiences. (b) Initialized from Qwen-2.5-7B-Instruct, RetroAgent substantially outperforms the GRPO-trained baseline shao2024deepseekmath and achieves SOTA results across four challenging agentic benchmarks.
• Figure 2: Overview of the RetroAgent framework. After each episode, a self-reflection mechanism analyzes the trajectory to produce two forms of intrinsic feedback: $(\textup{\it i})$Intrinsic Numerical Feedback, which quantifies incremental subtask completion relative to prior attempts, rewarding promising exploratory behaviors that may not yet yield task success; and $(\textup{\it ii})$Intrinsic Language Feedback, which distills actionable lessons from past successes and failures into a memory buffer, retrieved via the proposed SimUtil-UCB strategy to effectively leverage accumulated experiences on similar tasks.
• Figure 3: Test-time adaptation in an in-distribution (ID) setting on WebShop and an out-of-distribution (OOD) setting on ALFWorld.
• Figure 4: Robustness to challenging tasks on MineSweeper.
• Figure 5: Accuracy of subtask completion scores generated via single-trajectory (single) vs. pairwise-trajectory (pairwise) induction for Qwen-2.5-7B-Instruct on WebShop.