Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planning

Min Zhang, Xian Fu, Jianye Hao, Peilong Han, Hao Zhang, Lei Shi, Hongyao Tang, Yan Zheng

TL;DR

This work attempts to provide an in-depth and comprehensive evaluation of the performance of MFM s on embodied task planning, aiming to shed light on their capabilities and limitations in this domain.

Abstract

In recent years, Multi-modal Foundation Models (MFMs) and Embodied Artificial Intelligence (EAI) have been advancing side by side at an unprecedented pace. The integration of the two has garnered significant attention from the AI research community. In this work, we attempt to provide an in-depth and comprehensive evaluation of the performance of MFM s on embodied task planning, aiming to shed light on their capabilities and limitations in this domain. To this end, based on the characteristics of embodied task planning, we first develop a systematic evaluation framework, which encapsulates four crucial capabilities of MFMs: object understanding, spatio-temporal perception, task understanding, and embodied reasoning. Following this, we propose a new benchmark, named MFE-ETP, characterized its complex and variable task scenarios, typical yet diverse task types, task instances of varying difficulties, and rich test case types ranging from multiple embodied question answering to embodied task reasoning. Finally, we offer a simple and easy-to-use automatic evaluation platform that enables the automated testing of multiple MFMs on the proposed benchmark. Using the benchmark and evaluation platform, we evaluated several state-of-the-art MFMs and found that they significantly lag behind human-level performance. The MFE-ETP is a high-quality, large-scale, and challenging benchmark relevant to real-world tasks.

MFE-ETP: A Comprehensive Evaluation Benchmark for Multi-modal Foundation Models on Embodied Task Planning

TL;DR

This work attempts to provide an in-depth and comprehensive evaluation of the performance of MFM s on embodied task planning, aiming to shed light on their capabilities and limitations in this domain.

Abstract

In recent years, Multi-modal Foundation Models (MFMs) and Embodied Artificial Intelligence (EAI) have been advancing side by side at an unprecedented pace. The integration of the two has garnered significant attention from the AI research community. In this work, we attempt to provide an in-depth and comprehensive evaluation of the performance of MFM s on embodied task planning, aiming to shed light on their capabilities and limitations in this domain. To this end, based on the characteristics of embodied task planning, we first develop a systematic evaluation framework, which encapsulates four crucial capabilities of MFMs: object understanding, spatio-temporal perception, task understanding, and embodied reasoning. Following this, we propose a new benchmark, named MFE-ETP, characterized its complex and variable task scenarios, typical yet diverse task types, task instances of varying difficulties, and rich test case types ranging from multiple embodied question answering to embodied task reasoning. Finally, we offer a simple and easy-to-use automatic evaluation platform that enables the automated testing of multiple MFMs on the proposed benchmark. Using the benchmark and evaluation platform, we evaluated several state-of-the-art MFMs and found that they significantly lag behind human-level performance. The MFE-ETP is a high-quality, large-scale, and challenging benchmark relevant to real-world tasks.
Paper Structure (28 sections, 13 figures, 8 tables)

This paper contains 28 sections, 13 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Evaluation Framework
  3. Evaluation Benchmark
  4. Data Collection
  5. Benchmark Structure
  6. Automatic Evaluation
  7. Data Preparation
  8. Model Delpoyment
  9. Metric Calculation
  10. Experiments
  11. Results on Embodied Q&A Tasks
  12. Results on Planning Q&A Tasks
  13. Conclusion
  14. Appendix
  15. Appendix Overview
  16. ...and 13 more sections

Figures (13)

  • Figure 1: The overall evaluation framework tailored for embodied task planning encapsulates four crucial capabilities: Object Understanding, Spatio-Temporal perception, Task Understanding, and Embodied Reasoning.
  • Figure 2: Example questions of MFE-ETP benchmark and illustration of the six types of task formats.
  • Figure 3: This figure outlines our evaluation workflow. It involves benchmark construction, data preparation, model deployment, and metric calculation to derive the final score.
  • Figure 4: Statistics of MFE-ETP benchmark.
  • Figure 5: Radar chart of MFMs' scores on MFE-ETP.
  • ...and 8 more figures