Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents

Zhili Cheng, Yuge Tu, Ran Li, Shiqi Dai, Jinyi Hu, Shengding Hu, Jiahao Li, Yang Shi, Tianyu Yu, Weize Chen, Lei Shi, Maosong Sun

TL;DR

EmbodiedEval introduces an interactive, large-scale benchmark for evaluating multimodal LLMs as embodied agents in 3D environments. It combines diverse, automatically generated tasks across five categories in 125 scenes within a unified evaluation framework, enabling end-to-end assessment of navigation, manipulation, social interaction, and reasoning. Experimental results reveal a substantial gap between current MLLMs and human performance, particularly on long-horizon and interaction-centric tasks, highlighting areas for improvement in grounding, spatial understanding, and planning. By open-sourcing the data and simulator, EmbodiedEval aims to accelerate progress toward truly embodied intelligence in multimodal models.

Abstract

Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.

EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents

TL;DR

EmbodiedEval introduces an interactive, large-scale benchmark for evaluating multimodal LLMs as embodied agents in 3D environments. It combines diverse, automatically generated tasks across five categories in 125 scenes within a unified evaluation framework, enabling end-to-end assessment of navigation, manipulation, social interaction, and reasoning. Experimental results reveal a substantial gap between current MLLMs and human performance, particularly on long-horizon and interaction-centric tasks, highlighting areas for improvement in grounding, spatial understanding, and planning. By open-sourcing the data and simulator, EmbodiedEval aims to accelerate progress toward truly embodied intelligence in multimodal models.

Abstract

Multimodal Large Language Models (MLLMs) have shown significant advancements, providing a promising future for embodied agents. Existing benchmarks for evaluating MLLMs primarily utilize static images or videos, limiting assessments to non-interactive scenarios. Meanwhile, existing embodied AI benchmarks are task-specific and not diverse enough, which do not adequately evaluate the embodied capabilities of MLLMs. To address this, we propose EmbodiedEval, a comprehensive and interactive evaluation benchmark for MLLMs with embodied tasks. EmbodiedEval features 328 distinct tasks within 125 varied 3D scenes, each of which is rigorously selected and annotated. It covers a broad spectrum of existing embodied AI tasks with significantly enhanced diversity, all within a unified simulation and evaluation framework tailored for MLLMs. The tasks are organized into five categories: navigation, object interaction, social interaction, attribute question answering, and spatial question answering to assess different capabilities of the agents. We evaluated the state-of-the-art MLLMs on EmbodiedEval and found that they have a significant shortfall compared to human level on embodied tasks. Our analysis demonstrates the limitations of existing MLLMs in embodied capabilities, providing insights for their future development. We open-source all evaluation data and simulation framework at https://github.com/thunlp/EmbodiedEval.
Paper Structure (44 sections, 28 figures, 7 tables, 1 algorithm)

This paper contains 44 sections, 28 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. EmbodiedEval
  4. Evaluation Formulation
  5. Task Categories
  6. Benchmark Construction
  7. Scene Collection
  8. Task Collection
  9. Task Annotation
  10. Dataset Statistics
  11. Experiments
  12. Experimental Setup
  13. Main Results
  14. Performance Analysis
  15. Error Case Analysis
  16. ...and 29 more sections

Figures (28)

  • Figure 1: Examples of the five task categories and performance overview of EmbodiedEval. The embodied agent powered by MLLMs is required to finish the given task in a 3D simulation environment.
  • Figure 2: The evaluation process of EmbodiedEval. The task description and ego-centric observation history will be input for the model. The environment will respond to the action from the model output with a new observation.
  • Figure 3: The dataset construction pipeline of EmbodiedEval .
  • Figure 4: Dataset statistics of EmbodiedEval. Left: Number of tasks by category for each scene source. Middle: Visualization of vocabulary by part of speech and word frequency.
  • Figure 5: Left: Success rate vs. number of steps required for the task. Middle: Success rate vs. allowed max steps. Right: The success rate of Gemini-Flash with different number of images as input across five task categories.
  • ...and 23 more figures