Structure Causal Models and LLMs Integration in Medical Visual Question Answering

TL;DR

This work addresses confounding biases in Medical Visual Question Answering by introducing a causal inference framework (CIF) that leverages front-door adjustment with two mediators to deconfound image-question interactions. A mediator-informed sampling strategy guided by mutual information, plus a prompt module (PM) that generates structured QA prompts, enables robust causal feature alignment and improves LLM-guided answer generation. The approach is instantiated with CLIP-based visual features and LLaMA-based language modeling, and evaluated across five MedVQA datasets, achieving consistent accuracy gains, notably on open-ended questions and more complex reasoning tasks. The combination of CIF and PM demonstrates a synergistic effect, reducing reliance on spurious correlations and enhancing generalization in multi-modal medical data, with practical implications for reliable clinical QA systems.

Abstract

Medical Visual Question Answering (MedVQA) aims to answer medical questions according to medical images. However, the complexity of medical data leads to confounders that are difficult to observe, so bias between images and questions is inevitable. Such cross-modal bias makes it challenging to infer medically meaningful answers. In this work, we propose a causal inference framework for the MedVQA task, which effectively eliminates the relative confounding effect between the image and the question to ensure the precision of the question-answering (QA) session. We are the first to introduce a novel causal graph structure that represents the interaction between visual and textual elements, explicitly capturing how different questions influence visual features. During optimization, we apply the mutual information to discover spurious correlations and propose a multi-variable resampling front-door adjustment method to eliminate the relative confounding effect, which aims to align features based on their true causal relevance to the question-answering task. In addition, we also introduce a prompt strategy that combines multiple prompt forms to improve the model's ability to understand complex medical data and answer accurately. Extensive experiments on three MedVQA datasets demonstrate that 1) our method significantly improves the accuracy of MedVQA, and 2) our method achieves true causal correlations in the face of complex medical data.

Figures (6)

• Figure 1: Illustration of how the confounder C affects model inference. The imbalanced training set leads to data bias, which can manifest in various ways, such as (a) difficulty in learning pathology, (b) statistical association, (c) relativity of confounders, and (d) co-occurrence of multiple pathologies, among others. These biases introduce confounders during inference, creating misleading causal paths. As a result, the model relies on spurious correlations rather than true causal relationships, leading to incorrect predictions.
• Figure 2: Causal directed acyclic graph of MedVQA, gray variables are observed data, and blue variables are mediators. (a) Important variables in MedVQA and their associations. (b) We apply the front-door adjustment by introducing two mediators to deal with the invisible confounders. This reveals the true causal relationship between $\left \{I,Q\right \}$ and $A$. The dashed line between the two mediators reflects the relative confounders considering the interaction between the two modalities.
• Figure 3: Overview of our method. There are causal features, confounding features, and their spurious associations in the feature space. We introduce an indicator module to obtain true causal features in MedVQA data, where we use mutual information to adjust the factors that may lead to spurious associations when learning mediators. For the features of two modalities, we use a multi-variable resampling front-door adjustment method to separate causation and confounding factors. The prompt module receives a variety of inputs, including instruction, questions, and a new set of QA pairs generated from each original data, which are then fed into the pre-trained linguistic decoder as an overall prompt for answering.
• Figure 4: Overview of indicator structure. The indicator structure consists of a visual indicator, a text indicator, and a multi-modal feature fusion module. This structure generates two mediators, which are combined with $F_{i}$ and $F_{q}$ to obtain true causal features by front-door adjustment.
• Figure 5: Visualization Results of CIF on Lung (X-ray, CT), Brain (MRI), and Abdomen (CT) Imaging.