Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making medical vision-language models think causally across modalities with retrieval-augmented cross-modal reasoning

Weiqin Yang, Haowen Xue, Qingyi Peng, Hexuan Hu, Qian Huang, Tingbo Zhang

TL;DR

The paper tackles the tendency of medical vision-language models to rely on spurious correlations, which can yield non-factual outputs in high-stakes clinical settings. It introduces Multimodal Causal Retrieval-Augmented Generation (MCRAG), which integrates a Structural Causal Model (SCM) with a cross-modal causal graph to guide retrieval and grounding, including a causal factorization $p_G(V_D,V_F|V_I)=p(V_F|V_I)p(V_D|V_F)$. Key components comprise a two-stage graph construction (VLM-assisted discovery followed by manual refinement), domain-aware retrieval with causal scoring, and retrieval-augmented fine-tuning that enforces image grounding and causal coherence. Experiments on radiology VQA and report generation show state-of-the-art performance and ablations confirming the central role of causality, indicating that causal retrieval can substantially improve factuality, robustness, and interpretability. Overall, the work proposes a scalable path toward safer, more trustworthy medical VLMs by grounding cross-modal reasoning in explicit causal structure.

Abstract

Medical vision-language models (VLMs) achieve strong performance in diagnostic reporting and image-text alignment, yet their underlying reasoning mechanisms remain fundamentally correlational, exhibiting reliance on superficial statistical associations that fail to capture the causal pathophysiological mechanisms central to clinical decision-making. This limitation makes them fragile, prone to hallucinations, and sensitive to dataset biases. Retrieval-augmented generation (RAG) offers a partial remedy by grounding predictions in external knowledge. However, conventional RAG depends on semantic similarity, introducing new spurious correlations. We propose Multimodal Causal Retrieval-Augmented Generation, a framework that integrates causal inference principles with multimodal retrieval. It retrieves clinically relevant exemplars and causal graphs from external sources, conditioning model reasoning on counterfactual and interventional evidence rather than correlations alone. Applied to radiology report generation, diagnosis prediction, and visual question answering, it improves factual accuracy, robustness to distribution shifts, and interpretability. Our results highlight causal retrieval as a scalable path toward medical VLMs that think beyond pattern matching, enabling trustworthy multimodal reasoning in high-stakes clinical settings.

Making medical vision-language models think causally across modalities with retrieval-augmented cross-modal reasoning

TL;DR

The paper tackles the tendency of medical vision-language models to rely on spurious correlations, which can yield non-factual outputs in high-stakes clinical settings. It introduces Multimodal Causal Retrieval-Augmented Generation (MCRAG), which integrates a Structural Causal Model (SCM) with a cross-modal causal graph to guide retrieval and grounding, including a causal factorization . Key components comprise a two-stage graph construction (VLM-assisted discovery followed by manual refinement), domain-aware retrieval with causal scoring, and retrieval-augmented fine-tuning that enforces image grounding and causal coherence. Experiments on radiology VQA and report generation show state-of-the-art performance and ablations confirming the central role of causality, indicating that causal retrieval can substantially improve factuality, robustness, and interpretability. Overall, the work proposes a scalable path toward safer, more trustworthy medical VLMs by grounding cross-modal reasoning in explicit causal structure.

Abstract

Medical vision-language models (VLMs) achieve strong performance in diagnostic reporting and image-text alignment, yet their underlying reasoning mechanisms remain fundamentally correlational, exhibiting reliance on superficial statistical associations that fail to capture the causal pathophysiological mechanisms central to clinical decision-making. This limitation makes them fragile, prone to hallucinations, and sensitive to dataset biases. Retrieval-augmented generation (RAG) offers a partial remedy by grounding predictions in external knowledge. However, conventional RAG depends on semantic similarity, introducing new spurious correlations. We propose Multimodal Causal Retrieval-Augmented Generation, a framework that integrates causal inference principles with multimodal retrieval. It retrieves clinically relevant exemplars and causal graphs from external sources, conditioning model reasoning on counterfactual and interventional evidence rather than correlations alone. Applied to radiology report generation, diagnosis prediction, and visual question answering, it improves factual accuracy, robustness to distribution shifts, and interpretability. Our results highlight causal retrieval as a scalable path toward medical VLMs that think beyond pattern matching, enabling trustworthy multimodal reasoning in high-stakes clinical settings.
Paper Structure (10 sections, 2 equations, 1 figure, 3 tables)

This paper contains 10 sections, 2 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Structural Causal Model (SCM)
  4. Cross-Modal Medical Causal Graph Construction
  5. Causal-based Retrieval Augmented Reasoning
  6. Experiment
  7. Experimental Setups
  8. Comparison Results
  9. Limitations
  10. Conclusion

Figures (1)

  • Figure 1: MCRAG overview. Left (Graph Construction): A VLM extracts entities and relations from paired images and reports to construct a causal graph, followed by manual refinement to prune spurious links. Right (Retrieval and Generation): For a test image, the VLM queries the causal graph to retrieve top-k relevant reports ranked by a causal score. The retrieved reports and test image are then combined into a prompt for the generator VLM, which produces the final diagnosis.