Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards

Ziyu Chen, Zhiqing Xiao, Xinbei Jiang, Junbo Zhao

TL;DR

This work proposes a method that extracts experience from interaction records to model the underlying laws of the game environment, using these experience as internal motivation to guide agents.

Abstract

Large Language Models (LLMs) and Reinforcement Learning (RL) are two powerful approaches for building autonomous agents. However, due to limited understanding of the game environment, agents often resort to inefficient exploration and trial-and-error, struggling to develop long-term strategies or make decisions. We propose a method that extracts experience from interaction records to model the underlying laws of the game environment, using these experience as internal motivation to guide agents. These experience, expressed in language, are highly flexible and can either assist agents in reasoning directly or be transformed into rewards for guiding training. Our evaluation results in Crafter demonstrate that both RL and LLM agents benefit from these experience, leading to improved overall performance.

From Laws to Motivation: Guiding Exploration through Law-Based Reasoning and Rewards

TL;DR

This work proposes a method that extracts experience from interaction records to model the underlying laws of the game environment, using these experience as internal motivation to guide agents.

Abstract

Large Language Models (LLMs) and Reinforcement Learning (RL) are two powerful approaches for building autonomous agents. However, due to limited understanding of the game environment, agents often resort to inefficient exploration and trial-and-error, struggling to develop long-term strategies or make decisions. We propose a method that extracts experience from interaction records to model the underlying laws of the game environment, using these experience as internal motivation to guide agents. These experience, expressed in language, are highly flexible and can either assist agents in reasoning directly or be transformed into rewards for guiding training. Our evaluation results in Crafter demonstrate that both RL and LLM agents benefit from these experience, leading to improved overall performance.

Paper Structure

This paper contains 18 sections, 7 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Constrained Markov Decision Process
  5. Generate $\mathcal{E}$
  6. LLMs Reasoning with $\mathcal{E}$
  7. Reward Design with $\mathcal{E}$
  8. Experiments
  9. Conclusions and Limitations
  10. Related Works
  11. Starting from Records
  12. Crafter Details
  13. Implementation Details
  14. Data Collection
  15. Interaction Records
  16. ...and 3 more sections

Figures (4)

  • Figure 1: From Laws to Motivation. Experience approximates the laws of the environment and, through textual or reward, encourages the agents to achieve self-motivation.
  • Figure 2: Each observation in Crafter is a $9 \times 9$ local map, and the entire world is generated by randomly combining various types of grids according to certain rules.
  • Figure 3: Crafter defines 22 achievements that can be unlocked in each episode of gameplay. The arrows represent the dependencies between achievements, indicating that the achievement being pointed to is one of the prerequisites for completing the achievement it points towards.
  • Figure : Law-based Reward Generation with LLMs