PARL: Prompt-based Agents for Reinforcement Learning

TL;DR

PARL (Prompt-based Agent for Reinforcement Learning) is introduced, a method that uses LLMs as RL agents through prompting, without any fine-tuning, and can match or outperform traditional RL agents in simple environments by leveraging pretrained knowledge.

Abstract

Large language models (LLMs) have demonstrated high performance on tasks expressed in natural language, particularly in zero- or few-shot settings. These are typically framed as supervised (e.g., classification) or unsupervised (e.g., clustering) problems. However, limited work evaluates LLMs as agents in reinforcement learning (RL) tasks (e.g., playing games), where learning occurs through interaction with an environment and a reward system. While prior work focused on representing tasks that rely on a language representation, we study structured, non-linguistic reasoning - such as interpreting positions in a grid world. We therefore introduce PARL (Prompt-based Agent for Reinforcement Learning), a method that uses LLMs as RL agents through prompting, without any fine-tuning. PARL encodes actions, states, and rewards in the prompt, enabling the model to learn through trial-and-error interaction. We evaluate PARL on three standard RL tasks that do not entirely rely on natural language. We show that it can match or outperform traditional RL agents in simple environments by leveraging pretrained knowledge. However, we identify performance limitations in tasks that require complex mathematical operations or decoding states and actions.

Figures (8)

• Figure 1: Prompt-based Agent Optimization for Reinforcement Learning (PARL) uses a large language model to make decisions through in-context learning. The optimization begins with a prompt that contains only the task description. As the agent interacts with the environment, the prompt is updated by concatenating ($\oplus$) the history of interactions of the states, actions, and rewards generated at each step $t$.
• Figure 2: Example of a simplified prompt interaction for the Blackjack task, over three steps with the environment, equivalent to two episodes.
• Figure 3: Average reward during training with the PARL agent using LLM self-decoding of states. A smoothing window of five episodes is applied.
• Figure 4: Average reward during training with the PARL agent using script-based state decoding. A smoothing window of five episodes is applied.
• Figure 5: Comparison of average training rewards for SOTA agents (smoothing window = 200).