EmbodiedBrain: Expanding Performance Boundaries of Task Planning for Embodied Intelligence

TL;DR

EmbodiedBrain tackles the gap between LLMs and embodied agents by introducing a vision-language foundation model with an agent-aligned data format and Step-GRPO training. It integrates supervised fine-tuning with reinforcement learning guided by a Generative Reward Model and a rejection-sampling data curation pipeline, targeting long-horizon planning and robust spatial perception. A three-part evaluation suite (General, Planning, End-to-End Simulation) and a novel VLM-PlanSim-99 environment validate state-of-the-art performance across 14 benchmarks, and all data and weights are open-sourced. This work advances generalist embodied agents and provides a scalable framework for embodied task planning in real-world-like settings.

Abstract

The realization of Artificial General Intelligence (AGI) necessitates Embodied AI agents capable of robust spatial perception, effective task planning, and adaptive execution in physical environments. However, current large language models (LLMs) and multimodal LLMs (MLLMs) for embodied tasks suffer from key limitations, including a significant gap between model design and agent requirements, an unavoidable trade-off between real-time latency and performance, and the use of unauthentic, offline evaluation metrics. To address these challenges, we propose EmbodiedBrain, a novel vision-language foundation model available in both 7B and 32B parameter sizes. Our framework features an agent-aligned data structure and employs a powerful training methodology that integrates large-scale Supervised Fine-Tuning (SFT) with Step-Augumented Group Relative Policy Optimization (Step-GRPO), which boosts long-horizon task success by integrating preceding steps as Guided Precursors. Furthermore, we incorporate a comprehensive reward system, including a Generative Reward Model (GRM) accelerated at the infrastructure level, to improve training efficiency. For enable thorough validation, we establish a three-part evaluation system encompassing General, Planning, and End-to-End Simulation Benchmarks, highlighted by the proposal and open-sourcing of a novel, challenging simulation environment. Experimental results demonstrate that EmbodiedBrain achieves superior performance across all metrics, establishing a new state-of-the-art for embodied foundation models. Towards paving the way for the next generation of generalist embodied agents, we open-source all of our data, model weight, and evaluating methods, which are available at https://zterobot.github.io/EmbodiedBrain.github.io.

Figures (9)

• Figure 1: The Architecture of EmbodiedBrain: The model processes diverse multimodal inputs, including images of any resolution, long video sequences, and complex language instructions. Visual inputs are handled by a vision encoder and an MLP projector, while textual inputs are tokenized. All inputs are fed into a core LLM decoder, which performs deep reasoning and generates a structured output. The final output consists of three parts: a natural language response (<response>), a step-by-step plan (<plans>), and a sequence of executable actions (<actions>), enabling direct control and interaction within an embodied environment.
• Figure 2: Overview of training data of EmbodiedBrain
• Figure 3: Overall Data Distribution in the SFT Stage and the Distribution of Planning Data for Each Action
• Figure 4: Detailed process of the proposed Step-GRPO
• Figure 5: The three-stage pipeline for VLM-based task planning assessment (left) takes a natural language description and a scene image as input, processes it through Inference, Parsing, and Simulation stages, and outputs quantitative performance metrics. The Stage C Task Verification Framework (right) contains structured data input through adaptation, translation, and execution within the AI2-THOR environment, managed by a central execution engine.