Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspective

TL;DR

<3-5 sentence high-level summary>This work tackles unimodal biases in multimodal LLMs performing Visual Question Answering by introducing a causal framework to quantify language and vision biases. It builds MORE, a 12k MCQ dataset that demands multi-hop reasoning and directly tests bias resilience, and proposes CAVE, a causality-enhanced agent that decomposes questions, reasons with causal checks, and uses retrieval to verify answers. Across multiple MLLMs, the study finds strong unimodal biases and partial semantic understanding, with CAVE yielding meaningful improvements but not eliminating biases. Collectively, the work provides a rigorous causal lens, a challenging benchmark, and a practical bias-mitigation framework to advance robust, interpretable multimodal reasoning.

Abstract

Recent advancements in Large Language Models (LLMs) have facilitated the development of Multimodal LLMs (MLLMs). Despite their impressive capabilities, MLLMs often suffer from over-reliance on unimodal biases (e.g., language bias and vision bias), leading to incorrect answers or hallucinations in complex multimodal tasks. To investigate this issue, we propose a causal framework to interpret the biases in Visual Question Answering (VQA) problems. Within this framework, we conduct an in-depth causal analysis to assess the causal effect of these biases on MLLM predictions. Based on the analysis, we introduce 1) a novel MORE dataset with 12,000 challenging VQA instances requiring multi-hop reasoning and overcoming unimodal biases. 2) a causality-enhanced agent framework CAVE that guides models to comprehensively integrate information from different modalities and mitigate biases. Our experiments show that MLLMs perform poorly on MORE, indicating strong unimodal biases and limited semantic understanding. However, when integrated with our CAVE, promising improvements in reasoning and bias mitigation can be seen. These findings provide important insights for the development of more robust MLLMs and contribute to the broader goal of advancing multimodal AI systems capable of deeper understanding and reasoning. Our project page is at https://github.com/OpenCausaLab/MORE.

Figures (15)

• Figure 1: Examples of over-reliance on unimodal biases. MLLMs (e.g., LLaVA) erroneously generate answers due to language bias (indicated by the underlined text below the left image) and vision bias (the right image).
• Figure 2: Causal graph of MLLM's Prediction on VQA problems. We use the green subgraph $G_{h}$ to represent the desired causal mechanisms and compare it with the undesired effects of unimodal biases. We quantify the causal effects of each factor by performing controlled interventions of the images ($I, E, C$) and of the questions ($Q,T$).
• Figure 3: Our framework for generating data of MORE. We first prepare the image source and link the visual entity in a knowledge graph. Then, motivated by the visual and language bias analysis through the causal lens, we construct multiple-choice questions that require MLLMs to overcome unimodal biases and conduct multi-hop reasoning in a sampled subgraph. We also generate the causal (reasoning) rationale for each instance to provide interpretability.
• Figure 4: Question quality of MORE compared to other VQA datasets in terms of lexical diversity and fluency.
• Figure 5: Option distribution of MLLMs.