LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models

TL;DR

We address the rigidity of traditional RL policy exploration by introducing LLM-Explorer, a plug-in module that uses large language models to analyze the agent’s learning status and environment characteristics to dynamically generate task-specific exploration distributions. The method couples two LLMs in a two-stage workflow: one summarizes learning status from recent trajectories, and the other outputs a probability distribution (or action biases) to guide exploration for the next $K$ episodes, compatible with discrete and continuous action spaces. Empirical results on Atari and MuJoCo show substantial improvements (up to 37.27% mean human-normalized gain on Atari) with modest overhead (roughly $1.3 per run in API costs and ~10 hours per training). The work demonstrates broad compatibility across RL algorithms (DQN variants, DDPG, TD3) and several LLMs, highlighting the practicality and scalability of task-aware exploration in reinforcement learning. It also discusses limitations and safeguards, including handling skewed outputs, hallucinations, and potential adaptive strategies for broader deployment, while providing open-source code for reproducibility.

Abstract

Policy exploration is critical in reinforcement learning (RL), where existing approaches include greedy, Gaussian process, etc. However, these approaches utilize preset stochastic processes and are indiscriminately applied in all kinds of RL tasks without considering task-specific features that influence policy exploration. Moreover, during RL training, the evolution of such stochastic processes is rigid, which typically only incorporates a decay in the variance, failing to adjust flexibly according to the agent's real-time learning status. Inspired by the analyzing and reasoning capability of large language models (LLMs), we design LLM-Explorer to adaptively generate task-specific exploration strategies with LLMs, enhancing the policy exploration in RL. In our design, we sample the learning trajectory of the agent during the RL training in a given task and prompt the LLM to analyze the agent's current policy learning status and then generate a probability distribution for future policy exploration. Updating the probability distribution periodically, we derive a stochastic process specialized for the particular task and dynamically adjusted to adapt to the learning process. Our design is a plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants developed based on them. Through extensive experiments on the Atari and MuJoCo benchmarks, we demonstrate LLM-Explorer's capability to enhance RL policy exploration, achieving an average performance improvement up to 37.27%. Our code is open-source at https://github.com/tsinghua-fib-lab/LLM-Explorer for reproducibility.

Figures (9)

• Figure 1: Illustration of LLM-Explorer, which utilizes LLMs to generate task-specialized stochastic processes that can be dynamically adjusted to adapt to the learning process, enhancing policy exploration in RL.
• Figure 2: Performance of LLM-Explorer on the Atari and MuJoCo benchmark. In each experiment, we repeat with three different random seeds and the shaded areas indicate the standard deviations.
• Figure 3: Compatibility of LLM-Explorer with various RL algorithms. In each experiment, we repeat with three different random seeds and use the shaded area to indicate the standard deviations.
• Figure 4: Case study of the operating process of LLM-Explorer.
• Figure 5: Comparison among our LLM-Explorer design with baselines. In each experiment, we repeatedly run the training process with three different random seeds and use the shaded area to indicate the standard deviations.