EnvGen: Generating and Adapting Environments via LLMs for Training Embodied Agents

TL;DR

EnvGen tackles the inefficiency of using LLMs as ongoing agents by instead using them to generate and adapt training environments for a lightweight RL agent. The method operates in $N^{\text{Cycle}}$ cycles, where an LLM creates diverse environments, a small PPO-based agent learns in those settings, performance in the original environment is measured, and the LLM is fed per-task feedback to refine subsequent environments; only about $4$ LLM calls are used in total. Empirically, EnvGen improves Crafter performance beyond state-of-the-art baselines (e.g., achieving a score of $S$ around 32.2% vs 26-27% for strong baselines) and enables faster mastery of long-horizon tasks, while also stabilizing training in Heist. The approach demonstrates that adaptive, skill-targeted environment generation can yield significant practical gains for embodied RL, with much lower LLM utilization than direct-agent LLM methods, and it is supported by thorough ablations across update frequency, LLM choice, and curriculum strategies. $S = \exp\left(\frac{1}{22}\sum_{i=1}^{22}\ln(1+s_i)\right) - 1$ represents the geometric-mean score across Crafter achievements, reflecting robust performance across tasks.

Abstract

Recent SOTA approaches for embodied learning via interaction directly employ large language models (LLMs) as agents to determine the next steps in an environment. Due to their world knowledge and reasoning capabilities, LLM agents achieve stronger performance than previous smaller agents based on reinforcement learning (RL); however, frequently calling LLMs is slow and expensive. Instead of directly employing LLMs as agents, can we use LLMs' reasoning capabilities to adaptively create training environments to help smaller RL agents learn useful skills that they are weak at? We propose EnvGen, a novel framework to address this question. We first prompt an LLM to generate training environments by giving it the task description and simulator objectives that the agents should learn and then asking it to generate a set of environment configurations (e.g., different terrains, items initially given to agents, etc.). Next, we train a small RL agent in a mixture of the original and LLM-generated environments. Then, we enable the LLM to continuously adapt the generated environments to progressively improve the skills that the agent is weak at, by providing feedback to the LLM in the form of the agent's performance. We demonstrate the usefulness of EnvGen with comprehensive experiments in Crafter and Heist environments. We find that a small RL agent trained with EnvGen can outperform SOTA methods, including a GPT-4 agent, and learns long-horizon tasks significantly faster. We also show that using an LLM to adapt environments dynamically outperforms curriculum learning approaches and how the environments are adapted to help improve RL agents' weaker skills over time. Additionally, EnvGen is substantially more efficient as it only uses a small number of LLM calls (e.g., 4 in total), whereas LLM agents require thousands of calls. Lastly, we present detailed ablation studies for EnvGen design choices.

Figures (10)

• Figure 1: Comparison of different methods for creating embodied agents. Previous works commonly use (a) small RL agents or (b) LLM agents to explore skills. In (c) EnvGen, we train a small RL agent with diverse LLM-generated environments that train different skills in parallel and can be adapted via feedback to help the agents progressively improve skills that they are weaker at. Our method benefits from using the world knowledge from LLMs while maintaining efficient training through a lightweight RL agent.
• Figure 2: In EnvGen, we generate and adapt multiple training environments with an LLM to let the agent learn different skills effectively. EnvGen iterates over $N^{\text{Cycle}}$ cycles, each consisting of four steps (see \ref{['subsec:envgen']}).
• Figure 3: (a): Crafter gameplay screenshot. An agent explores a 2D world and completes 22 achievements. (b): Crafter achievement hierarchy. Some achievements can be completed right away; others require previous achievements to be unlocked first (i.e., in a hierarchical order following the arrows).
• Figure 4: Success rates for all the Crafter achievements of two PPO agents Moon2023AchievementDistillation -- (1) Baseline: trained in Crafter for 1.96M steps, and (2) Ours: trained in 0.96M steps in Crafter$^\text{EnvGen}$ and 1M in Crafter.
• Figure 5: Unlock times (the first moment when the agent completed an achievement) for three long-horizon achievements ('make stone pickaxe', 'make iron pickaxe', and 'make iron sword') of two PPO agents Moon2023AchievementDistillation -- (1) Baseline: trained in Crafter for 1.96M steps, and (2) Ours: trained for 0.96M steps in Crafter$^\text{EnvGen}$ and for 1M steps in Crafter. The plot shows the last 1M training steps out of 1.96M steps.