Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Models

TL;DR

Breast Cancer VLMs tackles the gap between high-performance multimodal modeling and clinical deployability in mammography. It combines ConvNet-based image representations with structured metadata using a hierarchical tabular-to-text tokenization and multi-stage co-attention fusion at native $2K$ resolution, plus synthetic report augmentation. The approach yields stable, cross-dataset improvements in malignancy and calcification detection, outperforming unimodal baselines and ViT-based methods, and demonstrating robustness with minimal annotation overhead. This work provides a practical pathway for clinically viable vision-language CAD in breast cancer screening.

Abstract

Breast cancer remains the most commonly diagnosed malignancy among women in the developed world. Early detection through mammography screening plays a pivotal role in reducing mortality rates. While computer-aided diagnosis (CAD) systems have shown promise in assisting radiologists, existing approaches face critical limitations in clinical deployment - particularly in handling the nuanced interpretation of multi-modal data and feasibility due to the requirement of prior clinical history. This study introduces a novel framework that synergistically combines visual features from 2D mammograms with structured textual descriptors derived from easily accessible clinical metadata and synthesized radiological reports through innovative tokenization modules. Our proposed methods in this study demonstrate that strategic integration of convolutional neural networks (ConvNets) with language representations achieves superior performance to vision transformer-based models while handling high-resolution images and enabling practical deployment across diverse populations. By evaluating it on multi-national cohort screening mammograms, our multi-modal approach achieves superior performance in cancer detection and calcification identification compared to unimodal baselines, with particular improvements. The proposed method establishes a new paradigm for developing clinically viable VLM-based CAD systems that effectively leverage imaging data and contextual patient information through effective fusion mechanisms.

Figures (3)

• Figure 1: Overview of our proposed method: a vision-language training/inference pipeline to use standard ConvNets and LLMs to integrate multi-modal information using Co-Attention mechanism and joint feature representation learning.
• Figure 2: The effect of various tokenizers.
• Figure 3: The effect of the number of tokens.