Seeing Right but Saying Wrong: Inter- and Intra-Layer Refinement in MLLMs without Training

TL;DR

This work tackles the mismatch between internal visual understanding and final outputs in multimodal LLMs by introducing DualPD, a training-free decoding refinement comprising inter-layer attention trajectory contrastive logits and intra-layer head-wise information filtering. By selecting a basic layer with maximal attention shift and softly suppressing low-contribution attention heads, DualPD aligns the model’s evolving visual reasoning with its predictions, yielding consistent accuracy gains across LLaVA and Qwen-VL models on diverse multimodal benchmarks. The approach requires no retraining, improves robustness across architectures, and is supported by ablations, sensitivity analyses, and qualitative studies that confirm better visual grounding and spatial reasoning. Overall, DualPD offers a practical, generalizable method to mitigate the seeing-right-but-saying-wrong phenomenon in MLLMs, with potential for broader adoption in real-time decoding scenarios.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities across a variety of vision-language tasks. However, their internal reasoning often exhibits a critical inconsistency: although deeper layers may attend to the correct visual regions, final predictions are frequently misled by noisy attention from earlier layers. This results in a disconnect between what the model internally understands and what it ultimately expresses, a phenomenon we describe as seeing it right but saying it wrong. To address this issue, we propose DualPD, a dual-perspective decoding refinement strategy that enhances the visual understanding without any additional training. DualPD consists of two components. (1) The layer-wise attention-guided contrastive logits module captures how the belief in the correct answer evolves by comparing output logits between layers that exhibit the largest attention shift. (2) The head-wise information filtering module suppresses low-contribution attention heads that focus on irrelevant regions, thereby improving attention quality within each layer. Experiments conducted on both the LLaVA and Qwen-VL model families across multiple multimodal benchmarks demonstrate that DualPD consistently improves accuracy without training, confirming its effectiveness and generalizability. The code will be released upon publication.

Figures (4)

• Figure 1: Layer difference for logits and attention.
• Figure 2: Overview of our framework: inter-layer contrastive attention and intra-layer attention refinement. The model first identifies a basic layer by selecting the one with the maximum visual attention distance to the final layer. It then computes the contrastive logits between the two layers to capture visual information evolution. The resulting logits are used for output prediction. The MHSA module (left) performs selective head masking, suppressing low-attention heads to reduce noise.
• Figure 3: Sensitivity analysis of logit suppression by attention head importance.
• Figure 4: Static basic layer selection across datasets. The gray dashed line indicates the best accuracy achieved by static selection of a fixed basic layer. The red dashed line represents the accuracy of our method with dynamically selected basic layers based on inter-layer attention difference.