MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics?

TL;DR

The paper presents MMRo, the first diagnostic benchmark specifically designed to evaluate Multimodal LLMs in robotics, focusing on perception, planning, visual reasoning, and safety across 14 sub-domains. It combines real and synthetic imagery with 26,175 QA pairs (open-ended and multiple-choice) and uses both GPT-4V and manual review to assess model performance, including ground-truth visual grounding through GroundingDINO and SAM. Experimental results across 13 MLLMs show no model excels across all dimensions, with significant gaps in perception and safety, and notable differences between open-source and commercial systems. This work provides a rigorous, publicly accessible framework to drive the development of robotics-oriented cognitive cores for embodied agents.

Abstract

It is fundamentally challenging for robots to serve as useful assistants in human environments because this requires addressing a spectrum of sub-problems across robotics, including perception, language understanding, reasoning, and planning. The recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated their exceptional abilities in solving complex mathematical problems, mastering commonsense and abstract reasoning. This has led to the recent utilization of MLLMs as the brain in robotic systems, enabling these models to conduct high-level planning prior to triggering low-level control actions for task execution. However, it remains uncertain whether existing MLLMs are reliable in serving the brain role of robots. In this study, we introduce the first benchmark for evaluating Multimodal LLM for Robotic (MMRo) benchmark, which tests the capability of MLLMs for robot applications. Specifically, we identify four essential capabilities perception, task planning, visual reasoning, and safety measurement that MLLMs must possess to qualify as the robot's central processing unit. We have developed several scenarios for each capability, resulting in a total of 14 metrics for evaluation. We present experimental results for various MLLMs, including both commercial and open-source models, to assess the performance of existing systems. Our findings indicate that no single model excels in all areas, suggesting that current MLLMs are not yet trustworthy enough to serve as the cognitive core for robots. Our data can be found in https://mm-robobench.github.io/.

Figures (9)

• Figure 1: The examples of 14 scenarios in MMRo.
• Figure 2: The dataset acquisition procedure and evaluation of MLLMs for robotics.
• Figure 3: Experimental results of open-ended questions. Evaluations are conducted on five MLLMs with our proposed MMRo across 14 scenarios. Note that the score for the visual grounding task is evaluated by mIoU, while the remaining tasks are evaluated by accuracy.
• Figure 4: Experimental results of open-ended questions for 13 MLLMs.
• Figure 5: Experimental results of multiple-choice questions for 13 MLLMs.