Representation Learning with Semantic-aware Instance and Sparse Token Alignments

TL;DR

This work addresses limitations of conventional contrastive medical vision-language pre-training, where treating all unpaired samples as negatives can disrupt semantic structure. It introduces SISTA, a multi-level framework combining Semantics-aware Instance-level Alignment (SIA) and Sparse Token-level Alignment (STA) to align medical images with radiology reports at both the image-report and patch-word levels. By leveraging inter-report similarity to mitigate false negatives and employing sparse cross-attention to link salient image patches to diagnostic terms, SISTA delivers robust cross-modal and intra-modal representations. Empirical results on MIMIC-CXR-based benchmarks show state-of-the-art performance across classification, segmentation, and object detection, especially under limited supervision, highlighting improved data efficiency and transferability for medical imaging tasks.

Abstract

Medical contrastive vision-language pre-training (VLP) has demonstrated significant potential in improving performance on downstream tasks. Traditional approaches typically employ contrastive learning, treating paired image-report samples as positives and unpaired ones as negatives. However, in medical datasets, there can be substantial similarities between images or reports from different patients. Rigidly treating all unpaired samples as negatives, can disrupt the underlying semantic structure and negatively impact the quality of the learned representations. In this paper, we propose a multi-level alignment framework, Representation Learning with Semantic-aware Instance and Sparse Token Alignments (SISTA) by exploiting the semantic correspondence between medical image and radiology reports at two levels, i.e., image-report and patch-word levels. Specifically, we improve the conventional contrastive learning by incorporating inter-report similarity to eliminate the false negatives and introduce a method to effectively align image patches with relevant word tokens. Experimental results demonstrate the effectiveness of the proposed framework in improving transfer performance across different datasets on three downstream tasks: image classification, image segmentation, and object detection. Notably, our framework achieves significant improvements in fine-grained tasks even with limited labeled data. Codes and pre-trained models will be made available.

Figures (2)

• Figure 1: Overview of our proposed SISTA framework for medical image-report alignment. The framework includes an image encoder and a text encoder to capture multimodal representations from chest X-rays and their corresponding reports. Sparse Token Alignment (STA) aligns specific regions in the image with disease tokens via sparse cross-attention, creating token-level correspondence. Semantic-aware Instance-level Vision Alignment (SIVA) and Semantic-aware Instance-level Language Alignment (SILA) ensure consistent instance-level alignment between images and reports.
• Figure 2: Heatmap visualization of token-level correspondence, with red regions indicating areas of higher activation weights associated with word tokens.