Meta-Entity Driven Triplet Mining for Aligning Medical Vision-Language Models

TL;DR

MedTrim tackles image-text misalignment in chest X-ray multimodal learning by introducing meta-entity guided triplet mining. It uses an ontology-based extractor to pull disease classes, adjectival and directional descriptors from radiology reports, and a novel entity-aware sampling score to mine informative triplets. A multimodal triplet objective that combines within- and cross-modal alignment yields superior retrieval and zero-shot classification on public CXR datasets. The approach promises more reliable med-VLMs and could extend to other imaging modalities with richer pathology attributes.

Abstract

Diagnostic imaging relies on interpreting both images and radiology reports, but the growing data volumes place significant pressure on medical experts, yielding increased errors and workflow backlogs. Medical vision-language models (med-VLMs) have emerged as a powerful framework to efficiently process multimodal imaging data, particularly in chest X-ray (CXR) evaluations, albeit their performance hinges on how well image and text representations are aligned. Existing alignment methods, predominantly based on contrastive learning, prioritize separation between disease classes over segregation of fine-grained pathology attributes like location, size or severity, leading to suboptimal representations. Here, we propose MedTrim (Meta-entity-driven Triplet mining), a novel method that enhances image-text alignment through multimodal triplet learning synergistically guided by disease class as well as adjectival and directional pathology descriptors. Unlike common alignment methods that separate broad disease classes, MedTrim leverages structured meta-entity information to preserve subtle but clinically significant intra-class variations. For this purpose, we first introduce an ontology-based entity recognition module that extracts pathology-specific meta-entities from CXR reports, as annotations on pathology attributes are rare in public datasets. For refined sample selection in triplet mining, we then introduce a novel score function that captures an aggregate measure of inter-sample similarity based on disease classes and adjectival/directional descriptors. Lastly, we introduce a multimodal triplet alignment objective for explicit within- and cross-modal alignment between samples sharing detailed pathology characteristics. Our demonstrations indicate that MedTrim improves performance in downstream retrieval and classification tasks compared to state-of-the-art alignment methods.

Figures (7)

• Figure 1: Multimodal CXR data carry key attributes pertaining to disease class, and adjectival and directional descriptors of pathology, which can show varying degrees of similarity between data samples. a) Conventional representational learning based on disease classes renders it challenging to dissociate positive samples (with closely aligned attributes) and semi-hard negative samples (with partial alignment of adjectival and directional attributes), forcing undesired alignment between samples with dissimilar attributes. b) To improve alignment, MedTrim leverages a novel multimodal triplet learning framework that explicitly selects positive and semi-hard negative samples, guided by not only disease class but also adjectival and directional descriptors. (A: anchor sample, P: positive sample, H: semi-hard negative sample, N: negative sample, Check mark: accurate alignment, Cross: incorrect alignment.)
• Figure 2: a) Given a batch of multimodal CXR samples, MedTrim first deploys its OBER module to identify disease labels along with adjectival and directional descriptors of corresponding pathology from radiology reports. b) Using these pathology attributes as meta entities, MedTrim performs triplet mining to select positive and semi-hard negative samples via entity-based weighting with respect to the anchor sample. The embeddings of the selected triplets are extracted with transformer-based image and text encoders. For image-text alignment, the encoders are fine-tuned via a multimodal triplet alignment objective that synergizes within-modal and cross-modal alignment expressed over embedding vectors.
• Figure 3: Distribution of retrieval performance across disease classes. Results shown for (a) I2T, (b) T2I retrieval tasks for disease class. MedTrim was compared against Contrastive and Triplet variants (see legend).
• Figure 4: (a) Ablation study on the number of mined triplets. P@50 (%; left y-axis) in I2I, I2T, T2I, and T2T tasks are plotted as a function of the number of triplets, along with the training time (min; right y-axis). (b) Ablation study on the percentage of semi-hard negative samples. P@50 in I2I, I2T, T2I, and T2T tasks are plotted as a function of the percentage of semi-hard negative samples included in triplets.
• Figure 5: Representative results for T2I retrieval. Given a query CXR report, the top-ranked images retrieved by competing methods are displayed, along with their similarity scores to the ground-truth CXR image. For each method, meta-entities extracted from the reports of the retrieved images are highlighted (magenta: disease class, purple: adjectival descriptor, yellow: directional descriptor).