MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

TL;DR

This work introduces MME-Reasoning, a large, structured benchmark designed to evaluate inductive, deductive, and abductive reasoning in multimodal LLMs while minimizing perceptual and domain-knowledge biases. It provides a diverse data collection and annotation framework, plus multi-format evaluation protocols that employ rule-based and open-ended prompts. The experiments reveal broad limitations and reasoning-type imbalances in current models, with abductive reasoning being the bottleneck and longer reasoning chains offering limited gains due to diminishing returns. The study offers insights into dataset design, evaluation methodology, and directions for improving multimodal reasoning through new training paradigms and more robust reasoning strategies.

Abstract

Logical reasoning is a fundamental aspect of human intelligence and an essential capability for multimodal large language models (MLLMs). Despite the significant advancement in multimodal reasoning, existing benchmarks fail to comprehensively evaluate their reasoning abilities due to the lack of explicit categorization for logical reasoning types and an unclear understanding of reasoning. To address these issues, we introduce MME-Reasoning, a comprehensive benchmark designed to evaluate the reasoning ability of MLLMs, which covers all three types of reasoning (i.e., inductive, deductive, and abductive) in its questions. We carefully curate the data to ensure that each question effectively evaluates reasoning ability rather than perceptual skills or knowledge breadth, and extend the evaluation protocols to cover the evaluation of diverse questions. Our evaluation reveals substantial limitations of state-of-the-art MLLMs when subjected to holistic assessments of logical reasoning capabilities. Even the most advanced MLLMs show limited performance in comprehensive logical reasoning, with notable performance imbalances across reasoning types. In addition, we conducted an in-depth analysis of approaches such as ``thinking mode'' and Rule-based RL, which are commonly believed to enhance reasoning abilities. These findings highlight the critical limitations and performance imbalances of current MLLMs in diverse logical reasoning scenarios, providing comprehensive and systematic insights into the understanding and evaluation of reasoning capabilities.

Figures (52)

• Figure 1: Performance comparison between thinking and chat models on MME-Reasoning.
• Figure 2: Example of questions in MME-Reasoning which covers comprehensive reasoning types.
• Figure 3: The overall construction process of MME-Reasoning.
• Figure 4: Overview of MME-Reasoning.
• Figure 5: Evaluation of rule-based questions.