Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

TL;DR

RoboBench proposes a holistic benchmark to evaluate multimodal LLMs as embodied brains, spanning five cognitive dimensions, 14 capabilities, and 25 tasks with 6092 QA pairs, using diverse embodiments and real robotic data. It introduces a world‑simulation framework to assess long‑horizon planning feasibility, moving beyond symbolic task matching to embodied feasibility. Large‑scale experiments with 14 MLLMs reveal persistent gaps in implicit instruction grounding, spatiotemporal perception, cross‑scenario planning, affordance reasoning, and execution‑level failure analysis, underscoring that current models only exhibit superficial embodied capabilities. By providing a comprehensive evaluation scaffold and actionable insights, RoboBench aims to guide the development of more robust, generalizable embodied multimodal AI for robotic manipulation.

Abstract

Building robots that can perceive, reason, and act in dynamic, unstructured environments remains a core challenge. Recent embodied systems often adopt a dual-system paradigm, where System 2 handles high-level reasoning while System 1 executes low-level control. In this work, we refer to System 2 as the embodied brain, emphasizing its role as the cognitive core for reasoning and decision-making in manipulation tasks. Given this role, systematic evaluation of the embodied brain is essential. Yet existing benchmarks emphasize execution success, or when targeting high-level reasoning, suffer from incomplete dimensions and limited task realism, offering only a partial picture of cognitive capability. To bridge this gap, we introduce RoboBench, a benchmark that systematically evaluates multimodal large language models (MLLMs) as embodied brains. Motivated by the critical roles across the full manipulation pipeline, RoboBench defines five dimensions-instruction comprehension, perception reasoning, generalized planning, affordance prediction, and failure analysis-spanning 14 capabilities, 25 tasks, and 6092 QA pairs. To ensure realism, we curate datasets across diverse embodiments, attribute-rich objects, and multi-view scenes, drawing from large-scale real robotic data. For planning, RoboBench introduces an evaluation framework, MLLM-as-world-simulator. It evaluate embodied feasibility by simulating whether predicted plans can achieve critical object-state changes. Experiments on 14 MLLMs reveal fundamental limitations: difficulties with implicit instruction comprehension, spatiotemporal reasoning, cross-scenario planning, fine-grained affordance understanding, and execution failure diagnosis. RoboBench provides a comprehensive scaffold to quantify high-level cognition, and guide the development of next-generation embodied MLLMs. The project page is in https://robo-bench.github.io.

Figures (30)

• Figure 1: Overview of RoboBench We evaluates MLLMs as embodied brains across 5 dimensions, 14 subdimensions, and 25 tasks, with tasks color-coded by type (top left). These dimensions follow the embodied execution pipeline (bottom)—from understanding intent, perceiving the environment, planning and adapting actions, refining subgoals via affordances, diagnosing failures—capturing the core cognitive roles of System 2. Performance comparison (top right) highlights significant gaps among state-of-the-art MLLMs, with Gemini-2.5-Pro achieving the best results.
• Figure 2: Examples of RoboBench Our benchmark covers 5 dimensions, 14 capabilities, and 25 tasks. We design diverse question formats, including multiple-choice, planning, and point prediction.
• Figure 3: Dataset Construction Pipeline. RoboBench customizes data workflows for each dimension: orange for instruction, green for perception, blue for planning, purple for affordance, and red for reflection. Each workflow generally follows three stages—preprocessing, annotation with VLMs, detection models or human experts, and Q&A format generation.
• Figure 4: Dimension Distribution of RoboBench.
• Figure 5: Dataset statistics.