GenRL: Multimodal-foundation world models for generalization in embodied agents

TL;DR

GenRL presents multimodal-foundation world models (MFWM) that align a foundation vision-language model with a latent world-model space, enabling task grounding from vision or language prompts without language annotations. By connecting a VLM embedder to a discrete latent dynamics model via a connector-aligner pair, and training task behavior in imagination through trajectory matching with a cosine-based reward, GenRL achieves strong multi-task generalization across locomotion and manipulation tasks. The framework supports data-free policy learning, enabling new tasks to be learned entirely in imagination, and demonstrates robustness to variations in training data distribution. The work advances foundation-model–style generalization in embodied RL, highlighting the potential and current limitations of aligning multimodal priors with environment dynamics.

Abstract

Learning generalist embodied agents, able to solve multitudes of tasks in different domains is a long-standing problem. Reinforcement learning (RL) is hard to scale up as it requires a complex reward design for each task. In contrast, language can specify tasks in a more natural way. Current foundation vision-language models (VLMs) generally require fine-tuning or other adaptations to be adopted in embodied contexts, due to the significant domain gap. However, the lack of multimodal data in such domains represents an obstacle to developing foundation models for embodied applications. In this work, we overcome these problems by presenting multimodal-foundation world models, able to connect and align the representation of foundation VLMs with the latent space of generative world models for RL, without any language annotations. The resulting agent learning framework, GenRL, allows one to specify tasks through vision and/or language prompts, ground them in the embodied domain's dynamics, and learn the corresponding behaviors in imagination. As assessed through large-scale multi-task benchmarking in locomotion and manipulation domains, GenRL enables multi-task generalization from language and visual prompts. Furthermore, by introducing a data-free policy learning strategy, our approach lays the groundwork for foundational policy learning using generative world models. Website, code and data: https://mazpie.github.io/genrl/

Figures (13)

• Figure 1: Multimodal-foundation world models connect and align the video-language space of a foundation model with the latent space of a generative world model for reinforcement learning, requiring vision-only data. Our GenRL framework turns visual and/or language prompts into latent targets and learns to realize the corresponding behaviors by training in the world model's imagination.
• Figure 2: Overview of GenRL. The agent learns a multimodal-foundation world model that connects and aligns (a) the representation of a foundation VLM with the latent states of a generative world model. Given a certain task prompt, (b) the model allows embedding the task and translating into targets in the latent dynamics space, which the agent can learn to achieve by using RL in imagination.
• Figure 3: When training the connector on (a) the VLM's representation we can address the multimodality gap in multiple ways: (b) prior works adopt noise during the training of the connector, (c) we adopt an aligner network that learns to map points in proximity of the visual embedding close the corresponding embedding.
• Figure 4: Language-to-action generalization. Offline RL from language prompts on tasks that are not deliberately included in the training dataset. Performance averaged over 10 seeds and standard error was reported with black lines. Detailed results per task in Appendix \ref{['app:results']}.
• Figure 5: Video-to-action. GenRL allows grounding video prompts into the target environment's dynamics. It allows visualization of the model's interpretation of the prompts, using the decoder (top row), and it allows turning prompts into behaviors, leading to generally higher performance than other approaches. 10 seeds. Additional visualizations on the project website.