Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality

TL;DR

This paper addresses multimodal hallucinations in Multimodal Large Language Models by treating visual and language priors as confounders in the attention–output pathway. It introduces CausalMM, a structural causal modeling framework that uses back-door adjustment and counterfactual reasoning on visual attention $A_i$ and language attention $A_t$ to estimate and mitigate their causal impact on the output $O$, despite modality priors. The approach defines an explicit SCM with inputs, attentions, embeddings, and priors, and employs four counterfactual attention patterns (Random, Uniform, Reversed, Shuffled) to isolate modality-specific effects. Experimental results on VLind-Bench, MME, and POPE demonstrate substantial improvements over decoding-based baselines, with multimodal collaboration providing the strongest gains and ablations confirming robustness of the counterfactual design. CausalMM is proposed as a training-free, plug-and-play solution to balance modal priors and enhance reliable multimodal reasoning in practical MLLMs, with code available at the provided repository.

Abstract

Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM

Figures (23)

• Figure 1: The comparison of conventional hallucination mitigation paradigm (e.g., VCD) and our proposed CausalMM.
• Figure 2: Causal diagram of counterfactual reasoning. ❶ In vision-only counterfactual reasoning, we only intervene in visual attention (i.e., the attention of the visual encoder). ❷ In language-only counterfactual reasoning, we only intervene in the multi-head self-attention of LLM. ❸ In multimodal collaborative counterfactual reasoning, we intervene in both visual and language attention at the same time and obtain the sum of their collaborative causal effects.
• Figure 3: Scores of different methods on VLind-Bench.CausalMM method significantly improves the model's score on VLind-Bench.
• Figure 4: Result comparison of different categories on MME Benchmark across different methods. In most tasks, the scores obtained by CausalMM are higher than baselines, which verifies its effectiveness.
• Figure 5: Result comparison of perception and cognition views on MME Benchmark across different methods. In both perception and cognition dimensions, variants of CausalMM outperform the others.