From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons

TL;DR

This work introduces the Generalist Embodied Agent (GEA), a unified multimodal LLM-based policy that grounds across diverse embodiments using a learned continuous-action tokenizer. GEAs training combines Stage 1 supervised finetuning on 2.2 million embodied trajectories with Stage 2 online reinforcement learning (PPO) to improve robustness and error recovery, aided by LoRA fine-tuning. Empirical results show strong generalization across manipulation, navigation, video games, UI control, and planning benchmarks, with cross-domain data and online RL providing notable gains over specialist and prior generalist methods. The approach highlights the value of cross-domain data and online interaction for building broad, capable embodied agents and provides code and models for community use.

Abstract

We examine the capability of Multimodal Large Language Models (MLLMs) to tackle diverse domains that extend beyond the traditional language and vision tasks these models are typically trained on. Specifically, our focus lies in areas such as Embodied AI, Games, UI Control, and Planning. To this end, we introduce a process of adapting an MLLM to a Generalist Embodied Agent (GEA). GEA is a single unified model capable of grounding itself across these varied domains through a multi-embodiment action tokenizer. GEA is trained with supervised learning on a large dataset of embodied experiences and with online RL in interactive simulators. We explore the data and algorithmic choices necessary to develop such a model. Our findings reveal the importance of training with cross-domain data and online RL for building generalist agents. The final GEA model achieves strong generalization performance to unseen tasks across diverse benchmarks compared to other generalist models and benchmark-specific approaches.

Figures (8)

• Figure 1: The Generalist Embodied Agent (GEA) is a multimodal LLM-based agent that can complete tasks from natural language instructions across a variety of domains and embodiments spanning manipulation, planning, game playing, and UI control. A pretrained MLLM is finetuned with supervised finetuning (SFT) on a large dataset of embodied experiences. The final GEA model is then finetuned with reinforcement learning (RL). GEA achieves competitive results in generalization to unseen settings.
• Figure 2: GEA utilizes a pretrained MLLM together with a multi-embodiment action tokenizer to enable a generalist agent to operate across a wide range of domains, embodiments, and action spaces. GEA takes as input information about the embodiment and desired task with the embodiment prompt and instruction and the observation visuals (bottom). It produces a sequence of action tokens in the LLM vocabulary, which are decoded by the multi-embodiment action detokenizer into an action for the appropriate embodiment and action space.
• Figure 3: GEA training stages. First, a MLLM is adapted to GEA-Base by finetuning the entire MLLM with SFT on interactive data. Next, GEA-Base is finetuned jointly with online RL (PPO) and SFT on the original data with LoRA.
• Figure 4: Online learning in Habitat Pick. MLLM methods finetune LLaVA-OV while other methods finetune GEA-Base.
• Figure 5: Analyzing the impact of training GEA with different base MLLMs with different parameter counts.