Toward Efficient Exploration by Large Language Model Agents

TL;DR

This work demonstrates that large language models can be harnessed to explicitly implement a data-efficient RL algorithm, Posterior Sampling for Reinforcement Learning (PSRL), by delegating posterior updating, posterior sampling, and policy optimization to separate LLMs. By representing epistemic uncertainty as a textual posterior and using language-grounded environment proxies (e.g., Wordle, combination lock, RiverSwim), the approach preserves PSRL’s exploration guarantees while enabling exploration in natural-language tasks. Empirical results show robust exploration in bandit and deterministic language tasks, with performance sensitive to the LLM's capabilities and prompting, and improved exploration under more powerful models in stochastic settings. The paper also discusses limitations in scaling stochastic environments and investigates alternatives like Information-Directed Sampling (IDS) to further balance exploration and data efficiency, suggesting a promising path for integrating classic RL algorithms with evolving LLM capabilities for real-world decision making.

Abstract

A burgeoning area within reinforcement learning (RL) is the design of sequential decision-making agents centered around large language models (LLMs). While autonomous decision-making agents powered by modern LLMs could facilitate numerous real-world applications, such successes demand agents that are capable of data-efficient RL. One key obstacle to achieving data efficiency in RL is exploration, a challenge that we demonstrate many recent proposals for LLM agent designs struggle to contend with. Meanwhile, classic algorithms from the RL literature known to gracefully address exploration require technical machinery that can be challenging to operationalize in purely natural language settings. In this work, rather than relying on finetuning or in-context learning to coax LLMs into implicitly imitating a RL algorithm, we illustrate how LLMs can be used to explicitly implement an existing RL algorithm (Posterior Sampling for Reinforcement Learning) whose capacity for statistically-efficient exploration is already well-studied. We offer empirical results demonstrating how our LLM-based implementation of a known, data-efficient RL algorithm can be considerably more effective in natural language tasks that demand prudent exploration.

Figures (15)

• Figure 1: Abstractly, an RL algorithm is an ordered sequence of steps. Existing approaches for LLM agent design (top) orchestrate some number of LLMs to implicitly induce a RL algorithm. In contrast, this paper advocates for a novel agent design principle (bottom) whereby an existing RL algorithm is explicitly implemented by outsourcing individual steps to distinct LLMs.
• Figure 2: The PSRL algorithm with LLM subroutines of posterior sampling, optimal behavior with respect to a sample, and posterior updating shown. Dotted arrows show data flow.
• Figure 3: Examples of a posterior (top) and posterior sample (bottom) generated by our LLM-based PSRL in Wordle
• Figure 4: Cumulative regret curves for a 5-armed Bernoulli bandit.
• Figure 5: Cumulative regret curves for the combination lock environment. The vertical axis shows turns to identify the unlock code.