Creative Agents: Empowering Agents with Imagination for Creative Tasks

TL;DR

Creative Agents introduces imagination as a core capability for open-ended task solving in embodied environments, presenting an imaginator–controller framework to generate and realize diverse outcomes in Minecraft. By pairing textual imagination from LLMs or visual imagination from diffusion models with BC or VLM-based controllers, the approach demonstrates diverse, credible building constructions and new evaluation metrics based on GPT-4V. The study shows diffusion-based imagination paired with GPT-4V yielding the strongest performance, supported by human-aligned assessments, and provides open-source datasets and models to spur future research. This work lays groundwork for scalable, imagination-driven creativity in AI agents and highlights practical evaluation strategies for open-ended tasks.

Abstract

We study building embodied agents for open-ended creative tasks. While existing methods build instruction-following agents that can perform diverse open-ended tasks, none of them demonstrates creativity -- the ability to give novel and diverse task solutions implicit in the language instructions. This limitation comes from their inability to convert abstract language instructions into concrete task goals in the environment and perform long-horizon planning for such complicated goals. Given the observation that humans perform creative tasks with the help of imagination, we propose a class of solutions for creative agents, where the controller is enhanced with an imaginator that generates detailed imaginations of task outcomes conditioned on language instructions. We introduce several approaches to implementing the components of creative agents. We implement the imaginator with either a large language model for textual imagination or a diffusion model for visual imagination. The controller can either be a behavior-cloning policy learned from data or a pre-trained foundation model generating executable codes in the environment. We benchmark creative tasks with the challenging open-world game Minecraft, where the agents are asked to create diverse buildings given free-form language instructions. In addition, we propose novel evaluation metrics for open-ended creative tasks utilizing GPT-4V, which holds many advantages over existing metrics. We perform a detailed experimental analysis of creative agents, showing that creative agents are the first AI agents accomplishing diverse building creation in the survival mode of Minecraft. Our benchmark and models are open-source for future research on creative agents (https://github.com/PKU-RL/Creative-Agents).

Figures (7)

• Figure 1: Overview of creative agents for open-ended creative tasks. A creative agent consists of two components: an imaginator and a controller. Given a free-form language instruction describing the creative task, the imaginator first generates the imagination in the form of text/image by LLM with Chain-of-Thought (CoT)/diffusion model, then the controller fulfills the imagination by executing actions in the environment, leveraging the code generation capability of vision-language model (VLM) or a behavior-cloning (BC) policy learned from data. We implement three combinations of the imaginator and controller: ① CoT+GPT-4, ② Diffusion+GPT-4V, and ③ Diffusion+BC.
• Figure 2: Comparison of all variants of creative agents in Minecraft building creation. For each evaluation metric, the number denotes the average score of the best agent over the 20 tasks. Diffusion+GPT-4V performs relatively better than other variants.
• Figure 3: Evaluation results in Minecraft building creation. Left: The Elo Rating of all agents based on the evaluation of GPT-4V. Right: The average overall score of each agent in all test tasks evaluated by GPT-4V and humans.
• Figure 4: Examples of the language description, the generated visual imagination, and the created building of each variant of creative agents. Visual imagination generated by the diffusion model has great diversity, which is an important manifestation of creativity.
• Figure 5: An example of the questionnaires for human evaluation. Left: 1v1 comparison between different methods; Right: directly score the test sample.