Retrospex: Language Agent Meets Offline Reinforcement Learning Critic

TL;DR

Retrospex tackles the challenge of leveraging past experiences to improve LLM-based agents without overloading the LLM context. It introduces a two-stage process—imitation-based warm-up followed by offline RL retrospection to train a lightweight RL Critic, which is then used in a dynamic action rescoring scheme to blend LLM action likelihood with action values. The dynamic weighting adapts to task horizon, allowing the Critic to increasingly influence decisions on longer trajectories, while maintaining LLM-guided behavior early on. Across ScienceWorld, ALFWorld, and Webshop, Retrospex shows consistent performance improvements over strong baselines, while offering lower inference overhead than full LLM retraining methods. This approach demonstrates a practical pathway to integrate experience memories into LLM-powered agents without sacrificing scalability or generality.

Abstract

Large Language Models (LLMs) possess extensive knowledge and commonsense reasoning capabilities, making them valuable for creating powerful agents. However, existing LLM agent frameworks have not fully utilized past experiences for improvement. This work introduces a new LLM-based agent framework called Retrospex, which addresses this challenge by analyzing past experiences in depth. Unlike previous approaches, Retrospex does not directly integrate experiences into the LLM's context. Instead, it combines the LLM's action likelihood with action values estimated by a Reinforcement Learning (RL) Critic, which is trained on past experiences through an offline ''retrospection'' process. Additionally, Retrospex employs a dynamic action rescoring mechanism that increases the importance of experience-based values for tasks that require more interaction with the environment. We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments, demonstrating its advantages over strong, contemporary baselines.

Figures (5)

• Figure 1: Comparing different architectures for LLM-based Agents
• Figure 2: The training process of Retrospex includes two stages: 1) In the Warm-up stage, an imitation learning (IL) base agent is trained and used for experience collection; 2) In the Retrospection stage: Offline RL is used to train RL Critic from offline experiences. Here, $s*$ and $a*$ denote states and actions, respectively. In the retrospection stage, $s'$ and $a'$ indicate the following state and action.
• Figure 3: Dynamic Action Rescoring in Retrospex, where $t$ indicates the interaction step in the current trajectory.
• Figure 4: $\alpha(t)$ with different values of steps $t$
• Figure 5: The structure of Twin-Q.