Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis

TL;DR

The paper investigates implicit visual misunderstandings (IVMs) in multimodal large language models by analyzing how attention distributes across multiple images. By decoupling visual and textual modalities in the causal attention, the authors show that deeper layers tend to concentrate attention on the image linked to the correct answer, even when the model may still produce correct responses without fully understanding the visual content. They introduce the STME benchmark and the attention accuracy metric to quantify IVMs through internal attention patterns, demonstrating robustness to positional biases and cross-model generalizability. The work extends to finer granularity with patch-level metrics and explores cross-modal and unimodal extensions, offering a practical internal-mechanism view for evaluating and guiding the training of MLLMs in visual tasks.

Abstract

Recent advancements have enhanced the capability of Multimodal Large Language Models (MLLMs) to comprehend multi-image information. However, existing benchmarks primarily evaluate answer correctness, overlooking whether models genuinely comprehend the visual input. To address this, we define implicit visual misunderstanding (IVM), where MLLMs provide correct answers without fully comprehending the visual input. Through our analysis, we decouple the visual and textual modalities within the causal attention module, revealing that attention distribution increasingly converges on the image associated with the correct answer as the network layers deepen. This insight leads to the introduction of a scale-agnostic metric, \textit{attention accuracy}, and a novel benchmark for quantifying IVMs. Attention accuracy directly evaluates the model's visual understanding via internal mechanisms, remaining robust to positional biases for more reliable assessments. Furthermore, we extend our approach to finer granularities and demonstrate its effectiveness in unimodal scenarios, underscoring its versatility and generalizability.

Figures (21)

• Figure 1: Left: Example of explicit visual misunderstandings: OCR deficiency and hallucination. Right: Example of implicit visual misunderstandings: the model provides a correct answer but actually describes the second image (while the question pertains to the content of the first image).
• Figure 2: Overview of the Approach for Identifying Model-focused Image. (a) MLLM completes a caption-matching task; (b) the attention submatrix for multimodal interactions is extracted; (c) for each layer, attention factor values for each image are calculated, allowing identification of the layer-focused image; (d) finally, the model-focus image is determined using the three metrics.
• Figure 3: Top: On the left, MLLM answers a caption matching question with the correct answer and explanation. On the right, the model’s attention converges on the target image. Bottom: MLLM answers an object counting question, where the fourth image corresponds to the correct answer. Despite providing the correct answer, the model’s reasoning is incorrect, showing IVMs. The heatmap reveals that the model's attention converges on a wrong image.
• Figure 4: The M-LND metric demonstrates the best performance, with attention accuracy exceeding 95% on hard tasks and achieving an astonishing 100% on easy tasks. The accuracy obtained with all three metrics is significantly higher than the results from direct instructions.
• Figure 5: As the model scale increases, the attention accuracy also improves, with models of varying scales exhibiting particularly high attention accuracy on less challenging tasks. In contrast, answer accuracy does not follow the same trend. This indicates an enhancement in the model's visual capabilities, but due to constraints in the downstream LLM, no corresponding performance improvement is observed.

Theorems & Definitions (3)

• Definition 2.1
• Definition 2.2
• Definition 2.3