BEAR: Benchmarking and Enhancing Multimodal Language Models for Atomic Embodied Capabilities

TL;DR

BEAR delivers the first fine-grained benchmark for atomic embodied capabilities in multimodal LLMs, structuring tasks into 6 categories and 14 skills across 4,469 image–video–text questions and 13 data sources. It exposes persistent gaps in current MLLMs’ embodied reasoning, with proprietary models outperforming open-source ones and limited gains from chain-of-thought prompting or test-time scaling. To address these challenges, BEAR-Agent integrates vision tools, a semantic scene graph, and a knowledge base into a dialog-driven agent, achieving a 9.12 percentage-point absolute improvement on GPT-5 (17.5% relative) and notable gains across perception and planning. The study also demonstrates that improving embodied capabilities benefits embodied-task performance in simulated environments, signaling practical potential for robust, generalizable embodied agents in real-world settings.

Abstract

Embodied capabilities refer to a suite of fundamental abilities for an agent to perceive, comprehend, and interact with the physical world. While multimodal large language models (MLLMs) show promise as embodied agents, a thorough and systematic evaluation of their embodied capabilities remains underexplored, as existing benchmarks primarily focus on specific domains such as planning or spatial understanding. To bridge this gap, we introduce BEAR, a comprehensive and fine-grained benchmark that evaluates MLLMs on atomic embodied capabilities. BEAR comprises 4,469 interleaved image-video-text entries across 14 domains in 6 categories, including tasks from low-level pointing, trajectory understanding, spatial reasoning, to high-level planning. Extensive evaluation results of 20 representative MLLMs reveal their persistent limitations across all domains of embodied capabilities. To tackle the shortfall, we propose BEAR-Agent, a multimodal conversable agent that integrates pretrained vision models to strengthen MLLM perception, 3D understanding, and planning capabilities. It substantially enhances MLLM performance across diverse embodied capabilities on BEAR, yielding a 9.12% absolute gain and a relative improvement of 17.5% on GPT-5. Furthermore, our experiments indicate that improving MLLM embodied capabilities can benefit embodied tasks in simulated environments. Project website: https://bear-official66.github.io/

Figures (92)

• Figure 1: Overview of BEAR. We introduce BEAR, the first benchmark for evaluating MLLMs in embodied capabilities. It covers 6 categories and 14 atomic skills, comprising 4,469 interleaved image–video–text VQA samples curated from 13 diverse data sources and tailored to each category.
• Figure 2: Statistics, category distribution and evaluation radar map of the BEAR benchmark.
• Figure 3: Long-horizon category in BEAR. The long-horizon category features 35 episodes collected from simulation environment. Each episode is decomposed into skill-oriented steps originate from five core categories and 14 skills in BEAR, ranging from perception to planning. Details in Appendix \ref{['appendix_long_horizon_section']}.
• Figure 4: Performance comparison across model types and prompting strategies.
• Figure 5: (a) Performance with respect to model size. We report overall performance across 6 categories. (b) Performance with respect to frame number. We report average performance of Spatial Reasoning and Task Planning to assess the effect of frame count on model performance.