Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography

Yuexi Du, John Onofrey, Nicha C. Dvornek

TL;DR

This work tackles the challenge of visual-language pre-training for mammography under data scarcity and high-resolution, multi-view imaging. It introduces MaMA, a multi-view and multi-scale alignment framework that combines inter-view image-image contrastive learning, symmetric image-text alignment, a Symmetric Local Alignment module, and parameter-efficient tuning of a medical LLM, along with structured report generation and meta-information masking. Across EMBED and RSNA-Mammo, MaMA outperforms state-of-the-art medical CLIP baselines while using only 52% of the largest baseline model size, achieving strong BI-RADS and density predictions and robust out-of-domain cancer detection with interpretable local correspondences. These results demonstrate a practical, data-efficient pathway to leverage visual-language pre-training for mammography, with potential clinical impact for improving radiologist workflows and decision support.

Abstract

Contrastive Language-Image Pre-training (CLIP) demonstrates strong potential in medical image analysis but requires substantial data and computational resources. Due to these restrictions, existing CLIP applications in medical imaging focus mainly on modalities like chest X-rays that have abundant image-report data available, leaving many other important modalities underexplored. Here, we propose one of the first adaptations of the full CLIP model to mammography, which presents significant challenges due to labeled data scarcity, high-resolution images with small regions of interest, and class-wise imbalance. We first develop a specialized supervision framework for mammography that leverages its multi-view nature. Furthermore, we design a symmetric local alignment module to better focus on detailed features in high-resolution images. Lastly, we incorporate a parameter-efficient fine-tuning approach for large language models pre-trained with medical knowledge to address data limitations. Our multi-view and multi-scale alignment (MaMA) method outperforms state-of-the-art baselines for three different tasks on two large real-world mammography datasets, EMBED and RSNA-Mammo, with only 52% model size compared with the largest baseline. The code is available at https://github.com/XYPB/MaMA

Multi-View and Multi-Scale Alignment for Contrastive Language-Image Pre-training in Mammography

TL;DR

This work tackles the challenge of visual-language pre-training for mammography under data scarcity and high-resolution, multi-view imaging. It introduces MaMA, a multi-view and multi-scale alignment framework that combines inter-view image-image contrastive learning, symmetric image-text alignment, a Symmetric Local Alignment module, and parameter-efficient tuning of a medical LLM, along with structured report generation and meta-information masking. Across EMBED and RSNA-Mammo, MaMA outperforms state-of-the-art medical CLIP baselines while using only 52% of the largest baseline model size, achieving strong BI-RADS and density predictions and robust out-of-domain cancer detection with interpretable local correspondences. These results demonstrate a practical, data-efficient pathway to leverage visual-language pre-training for mammography, with potential clinical impact for improving radiologist workflows and decision support.

Abstract

Contrastive Language-Image Pre-training (CLIP) demonstrates strong potential in medical image analysis but requires substantial data and computational resources. Due to these restrictions, existing CLIP applications in medical imaging focus mainly on modalities like chest X-rays that have abundant image-report data available, leaving many other important modalities underexplored. Here, we propose one of the first adaptations of the full CLIP model to mammography, which presents significant challenges due to labeled data scarcity, high-resolution images with small regions of interest, and class-wise imbalance. We first develop a specialized supervision framework for mammography that leverages its multi-view nature. Furthermore, we design a symmetric local alignment module to better focus on detailed features in high-resolution images. Lastly, we incorporate a parameter-efficient fine-tuning approach for large language models pre-trained with medical knowledge to address data limitations. Our multi-view and multi-scale alignment (MaMA) method outperforms state-of-the-art baselines for three different tasks on two large real-world mammography datasets, EMBED and RSNA-Mammo, with only 52% model size compared with the largest baseline. The code is available at https://github.com/XYPB/MaMA
Paper Structure (15 sections, 4 equations, 3 figures, 6 tables)

This paper contains 15 sections, 4 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Method
  4. Structured Report Construction
  5. Multi-view VLP
  6. Symmetric Local Alignment (SLA)
  7. Overall Pre-training Target
  8. LLM with PEFT as Text Encoder
  9. Experiments
  10. Pre-training Setup
  11. Experimental Setup
  12. Results
  13. Ablation Experiments
  14. Discussion and Conclusion
  15. Acknowledgement

Figures (3)

  • Figure 1: Comparison of Visual-Language Contrastive Learning Frameworks. (a) CLIP radford2021learning style; (b) SLIP mu2022slip style; (c) Our proposed MaMA, which aligns image-image and image-text features, exploiting the multi-view nature of mammography.
  • Figure 2: Proposed Multi-view and Multi-scale (MaMA) VLP Framework. (a) We utilize the multi-view information of mammography to conduct symmetric image-image and image-text contrastive learning. (b) We localize the most relevant sentence for each image patch and the most relevant patch for each sentence and align these matched local features via symmetric local alignment.
  • Figure 3: Local Similarity Maps Overlaid on Mammograms. We visualize the learned local similarity map for the "Impressions" sentence on test mammograms from EMBED jeong2023emory for MM-MIL wang2023using, our method with only visual localization, and our full method. Heat maps are normalized to [0,1]. The first row shows mammograms from the same side but different view; the second row shows mammograms from the same view but different side. The white boxes denote the dataset-provided annotated ROIs jeong2023emory.