Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations

Vaibhav Jain, Gerrit Grossmann

TL;DR

This work tackles exploration in sparse-reward RL by leveraging LLMs as planning guides. It introduces a soft-guidance framework that injects LLM hints into augmented observations, allowing standard RL algorithms to learn when to follow or ignore guidance without hard constraints. Across three BabyAI tasks of increasing difficulty, the method yields up to 71% relative improvements in final success and up to 9× faster sample efficiency, with the benefits growing in harder environments. The approach demonstrates that LLM planning capabilities can accelerate RL training while preserving algorithmic flexibility, albeit with computational costs from frequent LLM queries that warrant further optimization.

Abstract

Reinforcement Learning (RL) agents often struggle in sparse-reward environments where traditional exploration strategies fail to discover effective action sequences. Large Language Models (LLMs) possess procedural knowledge and reasoning capabilities from text pretraining that could guide RL exploration, but existing approaches create rigid dependencies where RL policies must follow LLM suggestions or incorporate them directly into reward functions. We propose a framework that provides LLM-generated action recommendations through augmented observation spaces, allowing RL agents to learn when to follow or ignore this guidance. Our method leverages LLMs' world knowledge and reasoning abilities while maintaining flexibility through soft constraints. We evaluate our approach on three BabyAI environments of increasing complexity and show that the benefits of LLM guidance scale with task difficulty. In the most challenging environment, we achieve 71% relative improvement in final success rates over baseline. The approach provides substantial sample efficiency gains, with agents reaching performance thresholds up to 9 times faster, and requires no modifications to existing RL algorithms. Our results demonstrate an effective method for leveraging LLM planning capabilities to accelerate RL training in challenging environments.

Guiding Exploration in Reinforcement Learning Through LLM-Augmented Observations

TL;DR

This work tackles exploration in sparse-reward RL by leveraging LLMs as planning guides. It introduces a soft-guidance framework that injects LLM hints into augmented observations, allowing standard RL algorithms to learn when to follow or ignore guidance without hard constraints. Across three BabyAI tasks of increasing difficulty, the method yields up to 71% relative improvements in final success and up to 9× faster sample efficiency, with the benefits growing in harder environments. The approach demonstrates that LLM planning capabilities can accelerate RL training while preserving algorithmic flexibility, albeit with computational costs from frequent LLM queries that warrant further optimization.

Abstract

Reinforcement Learning (RL) agents often struggle in sparse-reward environments where traditional exploration strategies fail to discover effective action sequences. Large Language Models (LLMs) possess procedural knowledge and reasoning capabilities from text pretraining that could guide RL exploration, but existing approaches create rigid dependencies where RL policies must follow LLM suggestions or incorporate them directly into reward functions. We propose a framework that provides LLM-generated action recommendations through augmented observation spaces, allowing RL agents to learn when to follow or ignore this guidance. Our method leverages LLMs' world knowledge and reasoning abilities while maintaining flexibility through soft constraints. We evaluate our approach on three BabyAI environments of increasing complexity and show that the benefits of LLM guidance scale with task difficulty. In the most challenging environment, we achieve 71% relative improvement in final success rates over baseline. The approach provides substantial sample efficiency gains, with agents reaching performance thresholds up to 9 times faster, and requires no modifications to existing RL algorithms. Our results demonstrate an effective method for leveraging LLM planning capabilities to accelerate RL training in challenging environments.

Paper Structure

This paper contains 23 sections, 2 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs in Reinforcement Learning.
  4. LLMs for Planning.
  5. LLM-Guided Reinforcement Learning
  6. Problem Formulation
  7. LLM Hint Generation
  8. Training Procedure
  9. Experiments
  10. Environment Details.
  11. Results
  12. Quality of LLM Planning Hints.
  13. Impact on RL Training Performance.
  14. Hint Frequency Effects.
  15. Conclusion and Discussion
  16. ...and 8 more sections

Figures (3)

  • Figure 1: Environment visualization with ASCII encoding (top) and corresponding LLM reasoning chain-of-thought response (bottom). The LLM analyzes the spatial relationships and mission requirements to determine the optimal action.
  • Figure 2: Training curves showing win-rate progression for frequency f=5 across environments. Our approach (LLM-hints) demonstrate faster convergence and higher final performance, particularly in complex environments like PickupLoc.
  • Figure 3: Training curves showing mean return and win-rate progression for GoToObj with Subgoal-based hints from Oracle. This illustrates that even with perfect subgoal hints, the agent still does show improvement over the baseline.