On the Modeling Capabilities of Large Language Models for Sequential Decision Making

TL;DR

The capabilities of Large Language Models for reinforcement learning (RL) across a diversity of interactive domains are investigated and crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration.

Abstract

Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for complex sequential decision making problems. In this paper, we investigate the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) across a diversity of interactive domains. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly, by first generating reward models to train an agent with RL. Our results show that, even without task-specific fine-tuning, LLMs excel at reward modeling. In particular, crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration. Finally, in environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting, further broadening their utility in sequential decision-making tasks.

Figures (11)

• Figure 1: AI feedback as the highest performance across different reward models derived from LLMs tested. AI feedback, which is a preference-based method for deriving a reward model from an LLM generally outperforms other methods.
• Figure 2: a) Building a reward model more-readily solves RL tasks than using an LLM as an actor. LLM-policy only performs well in domains with coarse-grained actions while LLM feedback presents strong performance across the entire range of action granularities. b) LLMs have unreliable zero-shot understanding of the environment dynamics. While LLMs can be used to craft useful reward models, their failure as direct policies may be explained by their poor understanding of the action space and the transition function.
• Figure 3: Rewards learned through AI Feedback distribute rewards to key timesteps. By doing so, the problem of credit assignment, or learning from delayed rewards, is significantly reduced. Such distribution effectively shortens the horizon over which the RL algorithm must propagate credit through its update rule.
• Figure 4: LLM preferences correlate with value function preferences. The correlation between Bradley-Terry models trained from frozen LLM state preferences and value function preferences increases as the online policy improves in 3 different domains.
• Figure 5: By changing the prompt, LLMs can be steered to provide feedback that promotes exploration on NetHack. Additionally, to avoid degenerate solutions, preferences should be elicited in an online fashion and the reward function be non-Markovian.