System-Mediated Attention Imbalances Make Vision-Language Models Say Yes

TL;DR

The paper challenges the image-centric view of vision-language hallucination by showing that redundant late-layer system attention drives the yes-bias. Through causal redistribution of attention weights across modalities in the text decoder, they demonstrate substantial suppression of yes-bias across six yes/no benchmarks, with particularly strong gains on compositional tasks. The authors formalize a system-mediated framework and show that relying on coarse representations underlies the bias, explaining why system-to-text and system-to-image redistributions outperform image-centric mitigations. The findings advocate holistic attention interventions that reallocate cross-modal influence, with potential implications for broader VLMs and autoregressive systems.

Abstract

Vision-language model (VLM) hallucination is commonly linked to imbalanced allocation of attention across input modalities: system, image and text. However, existing mitigation strategies tend towards an image-centric interpretation of these imbalances, often prioritising increased image attention while giving less consideration to the roles of the other modalities. In this study, we evaluate a more holistic, system-mediated account, which attributes these imbalances to functionally redundant system weights that reduce attention to image and textual inputs. We show that this framework offers a useful empirical perspective on the yes-bias, a common form of hallucination in which VLMs indiscriminately respond 'yes'. Causally redistributing attention from the system modality to image and textual inputs substantially suppresses this bias, often outperforming existing approaches. We further present evidence suggesting that system-mediated attention imbalances contribute to the yes-bias by encouraging a default reliance on coarse input representations, which are effective for some tasks but ill-suited to others. Taken together, these findings firmly establish system attention as a key factor in VLM hallucination and highlight its potential as a lever for mitigation.

Figures (4)

• Figure 1: Percentage shift in simple accuracy against percentage of attention weights from each source modality proportionally redistributed to the remaining modalities across all decoder heads (i.e. globally), for all six paired-prompt benchmarks in our evaluation suite. We observe that the majority of these interventions are harmful and gains are not consistent across benchmarks. Despite not exhibiting the same types of gains elsewhere, MME patterns with the rest of the benchmarks as far as the harmfulness of global interventions is concerned.
• Figure 2: Global (i.e. applied across all decoder heads) graduated pairwise redistributions of attention weights between all possible combinations of source and recipient modalities in SugarCrepe show whole-model interventions to mostly harm performance. These soft-touch interventions redistribute 10%, 20% and 30% of weights from source to recipient at a time. The same trend is reflected across all six benchmarks studied here, but results are not shown owing to space constraints.
• Figure 3: Percentage shift in simple accuracy against percentage of attention weights from each source modality proportionally redistributed to the remaining modalities per quarter, for all six paired-prompt benchmarks in our evaluation suite. These plots demonstrate that for five of our six benchmarks, proportional redistribution of system attention in Q4 yields the largest gains. MME is the sole exception to this pattern.
• Figure 4: Per-quarter 100% pairwise redistributions of attention weights between all possible combinations of source and recipient modalities in SugarCrepe show whole-model interventions to mostly harm performance. The same trend is reflected across all six benchmarks studied here, but results are not shown owing to space constraints.