Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Visual Room 2.0: Seeing is Not Understanding for MLLMs

Haokun Li, Yazhou Zhang, Jizhi Ding, Qiuchi Li, Peng Zhang

TL;DR

The paper investigates whether state-of-the-art multimodal large language models genuinely understand what they see, revealing a persistent gap between perception and higher-level cognition. It formalizes the Visual Room argument and introduces Visual Room 2.0, a hierarchical benchmark spanning 17 perception and cognition tasks across three levels, with 350 samples and 2,100 questions. Evaluation of ten SoTA multimodal models shows perceptual competence outpacing cognitive reasoning (about an 8-point gap); cognition does not reliably improve even with perfect perception, though scaling benefits cognition. The work provides a testbed and paradigm to assess Seeing vs Understanding and highlights the need for architectures and training that support genuine understanding beyond perceptual accuracy.

Abstract

Can multi-modal large language models (MLLMs) truly understand what they can see? Extending Searle's Chinese Room into the multi-modal domain, this paper proposes the Visual Room argument: MLLMs may describe every visual detail precisely yet fail to comprehend the underlying emotions and intentions, namely seeing is not understanding. Building on this, we introduce \textit{Visual Room} 2.0, a hierarchical benchmark for evaluating perception-cognition alignment of MLLMs. We model human perceptive and cognitive processes across three levels: low, middle, and high, covering 17 representative tasks. The perception component ranges from attribute recognition to scene understanding, while the cognition component extends from textual entailment to causal and social reasoning. The dataset contains 350 multi-modal samples, each with six progressive questions (2,100 in total) spanning perception to cognition. Evaluating 10 state-of-the-art (SoTA) MLLMs, we highlight three key findings: (1) MLLMs exhibit stronger perceptual competence than cognitive ability (8.0\%$\uparrow$); (2) cognition appears not causally dependent on perception-based reasoning; and (3) cognition scales with model size, but perception does not consistently improve with larger variants. This work operationalizes Seeing $\ne$ Understanding as a testable hypothesis, offering a new paradigm from perceptual processing to cognitive reasoning in MLLMs. Our dataset is available at https://huggingface.co/datasets/LHK2003/PCBench.

Visual Room 2.0: Seeing is Not Understanding for MLLMs

TL;DR

The paper investigates whether state-of-the-art multimodal large language models genuinely understand what they see, revealing a persistent gap between perception and higher-level cognition. It formalizes the Visual Room argument and introduces Visual Room 2.0, a hierarchical benchmark spanning 17 perception and cognition tasks across three levels, with 350 samples and 2,100 questions. Evaluation of ten SoTA multimodal models shows perceptual competence outpacing cognitive reasoning (about an 8-point gap); cognition does not reliably improve even with perfect perception, though scaling benefits cognition. The work provides a testbed and paradigm to assess Seeing vs Understanding and highlights the need for architectures and training that support genuine understanding beyond perceptual accuracy.

Abstract

Can multi-modal large language models (MLLMs) truly understand what they can see? Extending Searle's Chinese Room into the multi-modal domain, this paper proposes the Visual Room argument: MLLMs may describe every visual detail precisely yet fail to comprehend the underlying emotions and intentions, namely seeing is not understanding. Building on this, we introduce \textit{Visual Room} 2.0, a hierarchical benchmark for evaluating perception-cognition alignment of MLLMs. We model human perceptive and cognitive processes across three levels: low, middle, and high, covering 17 representative tasks. The perception component ranges from attribute recognition to scene understanding, while the cognition component extends from textual entailment to causal and social reasoning. The dataset contains 350 multi-modal samples, each with six progressive questions (2,100 in total) spanning perception to cognition. Evaluating 10 state-of-the-art (SoTA) MLLMs, we highlight three key findings: (1) MLLMs exhibit stronger perceptual competence than cognitive ability (8.0\%); (2) cognition appears not causally dependent on perception-based reasoning; and (3) cognition scales with model size, but perception does not consistently improve with larger variants. This work operationalizes Seeing Understanding as a testable hypothesis, offering a new paradigm from perceptual processing to cognitive reasoning in MLLMs. Our dataset is available at https://huggingface.co/datasets/LHK2003/PCBench.

Paper Structure

This paper contains 25 sections, 4 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. The Visual Room 2.0 Benchmark
  4. Visual Room Argument
  5. Benchmark Construction
  6. Benchmark Analysis
  7. Task distribution analysis.
  8. Hierarchical correlation analysis.
  9. Experiments and Analysis
  10. Experimental Settings
  11. Evaluation and Metrics
  12. Main Results
  13. Cognitive Performance under Correct Perception
  14. Difficulty Gradient Analysis
  15. Effect of Model Scaling on Perception and Cognition
  16. ...and 10 more sections

Figures (11)

  • Figure 1: The watch problem.
  • Figure 2: Definition of high-, mid- and low-level perception and cognition.
  • Figure 3: An example of visual room 2.0 benchmark.
  • Figure 4: Fig. (a) represents Spearman Correlation between different levels, and (b) represents Pearson Correlation.
  • Figure 5: Effect of model scaling on perception and cognition.
  • ...and 6 more figures