RL-GPT: Integrating Reinforcement Learning and Code-as-policy

TL;DR

RL-GPT tackles the challenge of enabling LLMs to operate effectively in open-world embodied tasks by coupling a slow planning agent with a fast code-as-policy agent within a two-level hierarchy. The framework integrates an RL training pipeline as a tool, allowing some actions to be coded by the LLM while others are learned via PPO, with a critic guiding iterative improvements. A task planner handles complex long-horizon subtasks, and a two-loop iteration mechanism refines both agents through environment feedback. Empirical results in MineDojo show state-of-the-art performance on multiple tasks and successful diamond acquisition, demonstrating improved sample efficiency and the ability to codify higher-level actions that enhance RL learning. Overall, RL-GPT provides a practical pathway for combining LLM reasoning, code generation, and RL to advance embodied AI in complex environments.

Abstract

Large Language Models (LLMs) have demonstrated proficiency in utilizing various tools by coding, yet they face limitations in handling intricate logic and precise control. In embodied tasks, high-level planning is amenable to direct coding, while low-level actions often necessitate task-specific refinement, such as Reinforcement Learning (RL). To seamlessly integrate both modalities, we introduce a two-level hierarchical framework, RL-GPT, comprising a slow agent and a fast agent. The slow agent analyzes actions suitable for coding, while the fast agent executes coding tasks. This decomposition effectively focuses each agent on specific tasks, proving highly efficient within our pipeline. Our approach outperforms traditional RL methods and existing GPT agents, demonstrating superior efficiency. In the Minecraft game, it rapidly obtains diamonds within a single day on an RTX3090. Additionally, it achieves SOTA performance across all designated MineDojo tasks.

Figures (5)

• Figure 1: The overview of RL-GPT. After the optimization in an environment, LLMs agents obtain optimized coded actions, RL achieves an optimized neural network, and our RL-GPT gets both optimized coded actions and neural networks. Our framework integrates the coding parts and the learning parts.
• Figure 2: To learn a subtask, the LLM can generate environment configurations (task, observation, reward, and action space) to instantiate RL. In particular, by reasoning about the agent behavior to solve the subtask, the LLM generates code to provide higher-level actions in addition to the original environment actions, improving the sample efficiency for RL.
• Figure 3: Overview of RL-GPT. The overall framework consists of a slow agent (orange) and a fast agent (green). The slow agent decomposes the task and determines "which actions" to learn. The fast agent writes code and RL configurations for low-level execution.
• Figure 4: The two-loop iteration. We design a method to optimize both slow agent and fast agent with a critic agent.
• Figure 5: Demonstrations of how different agents learn to harvest a log. While both RL agent and LLM agent learn a single type of solution (RL or code-as-policy), our RL-GPT can reasonably decompose the task and correct how to learn each sub-action through the slow iteration process. RL-GPT decomposes the task into "find a tree" and "cut a log", solving the former with code generation and the latter with RL. After a few iterations, it learns to provide RL with a necessary high-level action (attack 20 times) and completes the task with a high success rate. Best viewed by zooming in.