ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning

TL;DR

ERA addresses the gap between large, costly vision-language models and compact VLMs in embodied tasks by introducing Embodied Prior Learning (EPL) to inject three principled priors, followed by online RL to refine policies. EPL leverages Trajectory-Augmented Priors, Environment-Anchored Priors, and External Knowledge Priors, then a turn-aware PPO RL pipeline with self-summarization, dense rewards, and turn-level advantage estimation to tackle long-horizon, sparse-reward tasks. Extensive experiments on EB-ALFRED and EB-Manipulation demonstrate ERA-3B surpassing prompting-based large models and prior training baselines, with strong generalization to unseen tasks and robust ablations validating the priors and RL designs. This work offers a scalable, data-curated framework for building capable, generalizable embodied agents from compact VLMs, advancing practical embodied AI systems.

Abstract

Recent advances in embodied AI highlight the potential of vision language models (VLMs) as agents capable of perception, reasoning, and interaction in complex environments. However, top-performing systems rely on large-scale models that are costly to deploy, while smaller VLMs lack the necessary knowledge and skills to succeed. To bridge this gap, we present \textit{Embodied Reasoning Agent (ERA)}, a two-stage framework that integrates prior knowledge learning and online reinforcement learning (RL). The first stage, \textit{Embodied Prior Learning}, distills foundational knowledge from three types of data: (1) Trajectory-Augmented Priors, which enrich existing trajectory data with structured reasoning generated by stronger models; (2) Environment-Anchored Priors, which provide in-environment knowledge and grounding supervision; and (3) External Knowledge Priors, which transfer general knowledge from out-of-environment datasets. In the second stage, we develop an online RL pipeline that builds on these priors to further enhance agent performance. To overcome the inherent challenges in agent RL, including long horizons, sparse rewards, and training instability, we introduce three key designs: self-summarization for context management, dense reward shaping, and turn-level policy optimization. Extensive experiments on both high-level planning (EB-ALFRED) and low-level control (EB-Manipulation) tasks demonstrate that ERA-3B surpasses both prompting-based large models and previous training-based baselines. Specifically, it achieves overall improvements of 8.4\% on EB-ALFRED and 19.4\% on EB-Manipulation over GPT-4o, and exhibits strong generalization to unseen tasks. Overall, ERA offers a practical path toward scalable embodied intelligence, providing methodological insights for future embodied AI systems.

Figures (11)

• Figure 1: (a) Overview of the ERA framework: Embodied Prior Learning (EPL) finetunes on diverse data sources to provide foundational knowledge, and online RL further improves the agent. (b) ERA (i.e, EPL+RL) boosts a 3B base model to surpass GPT-4o on hold-out evaluation sets.
• Figure 2: Illustration of Embodied Prior Learning (EPL). EPL leverages three data sources: Augmented trajectory priors, environment-anchored priors, and external knowledge priors.
• Figure 3: (a) Our agent framework, and (b) a comparison of advantage estimation in turn-level GAE and token-level GAE. $A_{0,0}, \ldots, A_{T,n}$, and $A_0, \ldots,A_T$ refer to Advantage values for each response token.
• Figure 4: Examples of ERA performing step-by-step reasoning and actions: (a) on EB-ALFRED, it identifies and reflects on earlier errors; (b) on EB-Manipulation, it accurately places the star into the correct slot of the shape sorter.
• Figure 5: Performance correlation between stage 1 (EPL) and stage 2 (EPL+RL) on both seen and unseen sets. The Pearson correlation coefficient is reported in the top-left corner.