ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts

TL;DR

ImagineBench introduces the first standardized benchmark for reinforcement learning with LLM-imaginary rollouts, uniting real environment data with synthetic trajectories generated by fine-tuned LLMs across diverse domains. By pairing rollouts with natural language instructions of varying difficulty, it enables systematic evaluation of language-conditioned policies and offline RL methods. Empirical results show that naive offline RL on hybrid data improves generalization to unseen tasks but falls far short of real-rollout training, underscoring the need for algorithms that better leverage imagined experiences. The work outlines future directions including improved offline methods, fast online adaptation, and multi-modal imagination, and provides open-source code to accelerate progress in embodied, imagination-enabled agents.

Abstract

A central challenge in reinforcement learning (RL) is its dependence on extensive real-world interaction data to learn task-specific policies. While recent work demonstrates that large language models (LLMs) can mitigate this limitation by generating synthetic experience (noted as imaginary rollouts) for mastering novel tasks, progress in this emerging field is hindered due to the lack of a standard benchmark. To bridge this gap, we introduce ImagineBench, the first comprehensive benchmark for evaluating offline RL algorithms that leverage both real rollouts and LLM-imaginary rollouts. The key features of ImagineBench include: (1) datasets comprising environment-collected and LLM-imaginary rollouts; (2) diverse domains of environments covering locomotion, robotic manipulation, and navigation tasks; and (3) natural language task instructions with varying complexity levels to facilitate language-conditioned policy learning. Through systematic evaluation of state-of-the-art offline RL algorithms, we observe that simply applying existing offline RL algorithms leads to suboptimal performance on unseen tasks, achieving 35.44% success rate in hard tasks in contrast to 64.37% of method training on real rollouts for hard tasks. This result highlights the need for algorithm advancements to better leverage LLM-imaginary rollouts. Additionally, we identify key opportunities for future research: including better utilization of imaginary rollouts, fast online adaptation and continual learning, and extension to multi-modal tasks. Our code is publicly available at https://github.com/LAMDA-RL/ImagineBench.

Figures (6)

• Figure 1: We benchmark the problem of RL with LLM-imaginary rollouts. The LLM is fine-tuned to generate imaginary rollouts, followed by RL policy training using real and imaginary rollouts.
• Figure 2: Overview of ImagineBench. Three key features of our benchmark: (1) datasets of both real and LLM-imaginary rollouts, (2) diverse domains of environments, and (3) natural language instructions divided into various task levels. Examples shown in the 'Datasets' panel are from CLEVR-Robot environment.
• Figure 3: Illustration of the generation of LLM-imaginary rollouts. The LLM is first fine-tuned with the environment data, and then prompted to generate the rollouts for novel tasks.
• Figure 4: Examples of the LLM-imaginary rollouts for novel goals. The figures are obtained by rendering the states in LLM-imaginary rollouts.
• Figure 5: Success rate bars of different methods on various levels of goals. The x-axis denotes the offline RL algorithm, and the y-axis denotes the success rate for completing various natural language goals. 'w/ IR' stands for training with imaginary rollouts. The success rate is calculated based on the average of the last five checkpoints, and the error bars stand for the half standard deviation over three random seeds.