LLM-Guided Diagnostic Evidence Alignment for Medical Vision-Language Pretraining under Limited Pairing

TL;DR

This work tackles the challenge of medical vision–language pretraining with limited paired data by shifting from global and local alignment to diagnostic-evidence–level alignment. It introduces LGDEA, which uses LLMs to extract diagnostic evidence from radiology reports and builds a shared diagnostic evidence space with learnable prototypes; lesion-level visual cues are projected into this space, enabling evidence-guided cross-modal learning. The framework leverages both paired supervision and abundant unpaired data through seed-edge propagation and higher-order relation inference, achieving improvements in phrase grounding, image–text retrieval, and zero-shot classification, and even rivaling methods pretrained with far more paired data. The approach promises more clinically meaningful representations and practical applicability in data-scarce medical settings. Overall, LGDEA demonstrates the value of evidence-centric learning for medical VLP and offers a scalable path to leverage unpaired data in critical diagnostic contexts.

Abstract

Most existing CLIP-style medical vision--language pretraining methods rely on global or local alignment with substantial paired data. However, global alignment is easily dominated by non-diagnostic information, while local alignment fails to integrate key diagnostic evidence. As a result, learning reliable diagnostic representations becomes difficult, which limits their applicability in medical scenarios with limited paired data. To address this issue, we propose an LLM-Guided Diagnostic Evidence Alignment method (LGDEA), which shifts the pretraining objective toward evidence-level alignment that is more consistent with the medical diagnostic process. Specifically, we leverage LLMs to extract key diagnostic evidence from radiology reports and construct a shared diagnostic evidence space, enabling evidence-aware cross-modal alignment and allowing LGDEA to effectively exploit abundant unpaired medical images and reports, thereby substantially alleviating the reliance on paired data. Extensive experimental results demonstrate that our method achieves consistent and significant improvements on phrase grounding, image--text retrieval, and zero-shot classification, and even rivals pretraining methods that rely on substantial paired data.

Figures (7)

• Figure 1: Motivation of LGDEA. Global and local alignment may overlook diagnostic evidence, whereas LGDEA aligns images and reports in a shared diagnostic evidence space.
• Figure 2: Overview of the proposed LGDEA framework. (a) LLMs extract diagnostic evidence from radiology reports, and both report evidence and lesion-level visual cues are projected into a shared diagnostic evidence space. (b) Under limited pairing, paired evidence links are used as seed edges to align report and image nodes, while report--report and image--image graphs propagate evidence-aware relations to leverage abundant unpaired reports and images for vision--language pretraining.
• Figure 3: Evidence-guided higher-order alignment under scarce paired supervision. Sparse paired links $\mathbf{Y}$ are grounded in a shared diagnostic evidence space and propagated over intra-modal evidence graphs to infer higher-order relations $\mathbf{P}$.
• Figure 4: Attention maps for two disease categories are visualized on the MS-CXR dataset, comparing LGDEA with GT. Red bounding boxes indicate the ground truth regions relevant to phrase grounding. Highlighted pixels correspond to higher activation weights, reflecting stronger associations between specific diagnostic terms and image regions.
• Figure 5: Attention maps for eight disease categories are visualized on the MS-CXR dataset, comparing LGDEA with nine baseline methods. Red bounding boxes indicate the ground truth regions relevant to phrase grounding. Highlighted pixels correspond to higher activation weights, reflecting stronger associations between specific diagnostic terms and image regions.