Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Language Guided Exploration for RL Agents in Text Environments

Hitesh Golchha, Sahil Yerawar, Dhruvesh Patel, Soham Dan, Keerthiram Murugesan

TL;DR

This work introduces Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE) to provide decision-level guidance to an RL agent (called EXPLORER) and observes that on ScienceWorld, a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.

Abstract

Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like $\textit{tabula rasa}$ reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE ) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al.,2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.

Language Guided Exploration for RL Agents in Text Environments

TL;DR

This work introduces Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE) to provide decision-level guidance to an RL agent (called EXPLORER) and observes that on ScienceWorld, a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.

Abstract

Real-world sequential decision making is characterized by sparse rewards and large decision spaces, posing significant difficulty for experiential learning systems like reinforcement learning (RL) agents. Large Language Models (LLMs), with a wealth of world knowledge, can help RL agents learn quickly and adapt to distribution shifts. In this work, we introduce Language Guided Exploration (LGE) framework, which uses a pre-trained language model (called GUIDE ) to provide decision-level guidance to an RL agent (called EXPLORER). We observe that on ScienceWorld (Wang et al.,2022), a challenging text environment, LGE outperforms vanilla RL agents significantly and also outperforms other sophisticated methods like Behaviour Cloning and Text Decision Transformer.
Paper Structure (20 sections, 2 equations, 1 figure, 5 tables)

This paper contains 20 sections, 2 equations, 1 figure, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. The LGE framework
  5. Explorer
  6. Guide
  7. Experiments and Results
  8. Evaluating the Guide
  9. Gold Action Rank (GAR):
  10. Relevant Set Recall (RSR):
  11. Mean Avg. Precision (MAP):
  12. Evaluating LGE
  13. Conclusion
  14. Limitations
  15. Appendix
  16. ...and 5 more sections

Figures (1)

  • Figure 1: The Language Guided Exploration (LGE) Framework: The Guide uses contrastive learning to produce a set of feasible action given the task description thereby reducing substantially the space of possible actions. The Explorer, an RL agent, then uses the set of actions provided by the Guide to learn a policy and pick a suitable action using it.