Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models

TL;DR

The paper tackles object hallucination in multimodal LLMs caused by dataset co-occurrence biases. It introduces a causality-driven disentanglement framework comprising a Causal-Driven Visual Projector and a Causal Intervention Module in the final LLM layer, enabling backdoor adjustment to decouple spurious correlations. Empirically, the approach reduces hallucinations across benchmarks while preserving or enhancing visual and multimodal comprehension, with visualizations showing improved separability of object representations. This scalable, end-to-end method offers practical reliability improvements for vision-language reasoning and can be adapted to other large-scale architectures.

Abstract

Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual understanding tasks, yet they often suffer from object hallucinations--generating descriptions of objects that are inconsistent with or entirely absent from the input. This issue is closely related to dataset biases, where frequent co-occurrences of objects lead to entangled semantic representations across modalities. As a result, models may erroneously activate object representations that are commonly associated with the input but not actually present. To address this, we propose a causality-driven disentanglement framework that mitigates hallucinations through causal intervention. Our approach includes a Causal-Driven Projector in the visual pathway and a Causal Intervention Module integrated into the final transformer layer of the language model. These components work together to reduce spurious correlations caused by biased training data. Experimental results show that our method significantly reduces hallucinations while maintaining strong performance on multiple multimodal benchmarks. Visualization analyses further confirm improved separability of object representations. The code is available at: https://github.com/IgniSavium/Causal-LLaVA

Figures (19)

• Figure 1: left: A hallucination example where an MLLM incorrectly describes non-existent objects (chairs and a coffee cup) based on co-occurrence patterns. middle: Representations of original LLaVA's final LLM Transformer layer. The dining table (green dot, id=67) is surrounded by its top-10 most frequently co-occurring objects (red dots). right: Disentangled representations in our framework, with previously clustered objects now drifting significantly apart.
• Figure 2: Original LLaVA average object visual representations PCA after Vision Encoder (left), Projector (middle), and LLM layer 5 (right). Green dots: "dining table"; Red dots: top-10 co-occurring objects.
• Figure 3: Original LLaVA average object visual hidden state distributions PCA in LLM layers 15 (left), 30 (middle), and 40 (right). Green dots: "dining table"; Red dots: top-10 co-occurring objects.
• Figure 4: Unified illustration of causal mechanisms in MLLMs. (a) Backdoor adjustment blocks confounding paths. (b) Causal relationships involving visual/textual confounders in MLLM processing. (c) Final deconfounded results.
• Figure 5: Illustration of our innovative disentangled MLLM model architecture.