Evaluating Reasoning Faithfulness in Medical Vision-Language Models using Multimodal Perturbations

TL;DR

The paper introduces a clinically grounded framework to evaluate faithfulness of medical vision-language models by perturbing both text and image evidence in chest X-ray VQA. It provides a dedicated TAIX-VQA dataset and a three-axis evaluation (clinical fidelity, causal attribution, confidence calibration), validated via a radiologist reader study. Across six VLMs, the study shows final answer accuracy and explanation quality can decouple, with textual cues having larger impact on explanations than visual cues and with attribution proving more domain-critical than fidelity in some cases. The work advances reliable deployment considerations by highlighting the limits of using attribution alone as grounding and by releasing datasets and code to enable broader, reproducible evaluation of clinically faithful reasoning in multimodal models.

Abstract

Vision-language models (VLMs) often produce chain-of-thought (CoT) explanations that sound plausible yet fail to reflect the underlying decision process, undermining trust in high-stakes clinical use. Existing evaluations rarely catch this misalignment, prioritizing answer accuracy or adherence to formats. We present a clinically grounded framework for chest X-ray visual question answering (VQA) that probes CoT faithfulness via controlled text and image modifications across three axes: clinical fidelity, causal attribution, and confidence calibration. In a reader study (n=4), evaluator-radiologist correlations fall within the observed inter-radiologist range for all axes, with strong alignment for attribution (Kendall's $τ_b=0.670$), moderate alignment for fidelity ($τ_b=0.387$), and weak alignment for confidence tone ($τ_b=0.091$), which we report with caution. Benchmarking six VLMs shows that answer accuracy and explanation quality can be decoupled, acknowledging injected cues does not ensure grounding, and text cues shift explanations more than visual cues. While some open-source models match final answer accuracy, proprietary models score higher on attribution (25.0% vs. 1.4%) and often on fidelity (36.1% vs. 31.7%), highlighting deployment risks and the need to evaluate beyond final answer accuracy.

Figures (8)

• Figure 1: Proposed Approach. (Left) We evaluate state-of-the-art general and medical VLMs under controlled text and image modifications. (Center) All experiments use our expert-annotated chest X-ray VQA dataset. (Right) We introduce an automatic evaluation framework that scores CoTs for clinical fidelity, causal attribution, and confidence calibration, and we validate all metrics in a radiologist reader study.
• Figure 2: Automatic Evaluation. We construct paired prompts with a baseline case and a controlled modification to the image or text (here: heatmap overlay (VB-HM)). For each prompt the VLM produces an answer and CoT. The CoT for the modified input is scored by an LLM evaluator to quantify clinical fidelity (grounded in a curated knowledge base), causal attribution, and confidence calibration.
• Figure 3: Leave-one-out agreement. Kendall’s $\tau_b$ between each radiologist and the consensus of the remaining raters, and between the evaluator and the human consensus on the test split.
• Figure 4: Mean metric scores for clinical fidelity (CF), causal attribution (CA), and confidence calibration (CC) per model for each modification. Flip (F) and non-flip (NF) results appear in adjacent columns for each modification. CC is shown in grey to indicate it is exploratory and excluded from rankings. TB and VB conditions are shown as aligned with the ground truth answer ($\ast$) and misleading/unaligned ($\dagger$) cases because they favor or contradict a specific answer. VH and VO highlight or remove information without implying an answer and therefore carry no alignment label.
• Figure S1: Image-based modifications. (Left) Heatmap overlay as in VB-HM (bias via heatmap) and VH-HM (highlight via heatmap), (center) bounding box as in VB-BB (bias via bounding box) and VH-BB (highlight via bounding box), (right) black box occlusion as in VO-BB (occlusion))