Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion

TL;DR

The paper addresses the challenge of creating effective control policies from single or few demonstrations by reframing policy learning as generating policy network parameters from behavior prompts. It introduces Make-An-Agent, which combines a policy-parameter autoencoder, contrastive behavior embeddings, and a conditional latent diffusion model to synthesize latent policy representations that the decoder converts into deployable policies. The method is trained on a large dataset of policy networks and trajectories and demonstrates strong generalization to unseen tasks, cross-domain robustness, and real-world deployment across manipulation and locomotion domains. This approach offers a new paradigm for policy synthesis that bypasses explicit reward modeling or gradient-based adaptation, enabling efficient one-shot or few-shot policy generation with robustness to noise and environmental variability.

Abstract

Can we generate a control policy for an agent using just one demonstration of desired behaviors as a prompt, as effortlessly as creating an image from a textual description? In this paper, we present Make-An-Agent, a novel policy parameter generator that leverages the power of conditional diffusion models for behavior-to-policy generation. Guided by behavior embeddings that encode trajectory information, our policy generator synthesizes latent parameter representations, which can then be decoded into policy networks. Trained on policy network checkpoints and their corresponding trajectories, our generation model demonstrates remarkable versatility and scalability on multiple tasks and has a strong generalization ability on unseen tasks to output well-performed policies with only few-shot demonstrations as inputs. We showcase its efficacy and efficiency on various domains and tasks, including varying objectives, behaviors, and even across different robot manipulators. Beyond simulation, we directly deploy policies generated by Make-An-Agent onto real-world robots on locomotion tasks. Project page: https://cheryyunl.github.io/make-an-agent/

Figures (12)

• Figure 1: Overview: In the inference process of policy parameter generation, conditioning on behavior embeddings from the agent's trajectory, the latent diffusion model denoises random noise into a latent parameter representation, which can then be reconstructed as a deployable policy using the autoencoder. The forward process for progressively noising the data is also conducted on the latent space after encoding policy parameters as latent representations.
• Figure 2: Autoencoder: Encoding policy parameters into the latent space and decoding latent parameter representations into policy networks.
• Figure 3: Contrastive behavior embeddings: Learning informative behavior embeddings from long trajectories with contrastive loss.
• Figure 4: Visualization of MetaWorld, Robosuite, and real quadrupedal locomotion.
• Figure 5: Evaluation of seen tasks with 5 random initializations on MetaWorld and Robosuite. Our method generate policies using 5/10/50/100 test trajectories. Baselines are finetuned/adapted by the same test trajectories. Results are averaged over training with 4 seeds.