FaNe: Towards Fine-Grained Cross-Modal Contrast with False-Negative Reduction and Text-Conditioned Sparse Attention

TL;DR

FaNe addresses false negatives and coarse cross-modal alignment in medical vision–language pre-training by integrating semantic-aware positive mining, text-conditioned sparse attention pooling, and hard-negative intra-modal contrastive learning. The semantic class division with adaptive normalization identifies latent positives and organizes negatives, while multi-positive global alignment supports many-to-many pairings with a sigmoid-based loss $L_{mp}$. Fine-grained alignment leverages a sparse attention mask to ground sentence-level cues to image regions, with local losses $L_{t2i}$ and $L_{i2t}$ forming $L_{tc}$, complemented by the sparsity regularizer $L_{spa}$. A hard-negative intra-modal loss $L_{hn}$ further strengthens intra-modal discrimination. Pre-trained on MIMIC-CXR v2, FaNe yields state-of-the-art results across five medical benchmarks for classification, detection, and segmentation, demonstrating robust, fine-grained, cross-modal understanding in clinical imaging tasks.

Abstract

Medical vision-language pre-training (VLP) offers significant potential for advancing medical image understanding by leveraging paired image-report data. However, existing methods are limited by Fa}lse Negatives (FaNe) induced by semantically similar texts and insufficient fine-grained cross-modal alignment. To address these limitations, we propose FaNe, a semantic-enhanced VLP framework. To mitigate false negatives, we introduce a semantic-aware positive pair mining strategy based on text-text similarity with adaptive normalization. Furthermore, we design a text-conditioned sparse attention pooling module to enable fine-grained image-text alignment through localized visual representations guided by textual cues. To strengthen intra-modal discrimination, we develop a hard-negative aware contrastive loss that adaptively reweights semantically similar negatives. Extensive experiments on five downstream medical imaging benchmarks demonstrate that FaNe achieves state-of-the-art performance across image classification, object detection, and semantic segmentation, validating the effectiveness of our framework.

Figures (5)

• Figure 1: Illustration of false-negative problem and fine-grained alignment. (a) CLIP-style methods simply treat unpaired text or image as negatives, easily resulting in false-negative pairs (i.e, semantically similar but from different reports). (b) CLIP-style methods capture only global alignment, while color-coded sentences reveal the need for fine-grained region-level alignment.
• Figure 2: Comparison of text-conditioned sparse attention pooling with existing methods. (a) CLIP aligns global image $\text{v}^{\text{g}}$ and text $\text{t}^{\text{g}}$ tokens. (b) SigLIP uses a learnable global token $\text{v}^{\text{g}}$ as query to pool local tokens $\text{v}^{\text{l}}$ via cross-attention. (c) FLAIR adopts text-conditioned pooling with $\text{t}^{\text{g}}$ as query to aggregate $\text{v}^{\text{l}}$ for local alignment. (d) FaNe applies sparse attention for local text-aware features and employs a hard-negative aware contrastive loss $\mathcal{L}_{\text{hn}}$ to separate hard negatives for fine-grained understanding.
• Figure 3: Overview of FaNe. (a) A semantic similarity matrix is generated via a knowledge encoder (e.g., BioClinicalBERT me:biobert), enabling image-text pairs to be classified by a similarity threshold, with negatives further divided into hard and easy cases. (b) Text-conditioned features are derived through sparse attention, enabling alignment with sentence-level representations. (c) The combined losses $\mathcal{L}_{mp}$ and $\mathcal{L}_{hn}$ shape the feature space, where longer arrows indicate stronger repulsion for fine-grained concept learning. Sentences with the same highlight color share semantic meaning.
• Figure 4: Representative cross-modality attention maps. (a) The related sentence is "Mild thoracic scoliosis noted." (b) The related sentence is "Heart size is normal."
• Figure 5: Representative cross-modality attention maps by ResNet-50. (a) The related sentence is "The heart size remains mildly enlarged and status post aortic stent graft repair of a descending thoracic aortic dissection." (b) The related sentence is "There is stable minimal widening of the ascending aorta."