A Large Model for Non-invasive and Personalized Management of Breast Cancer from Multiparametric MRI

TL;DR

A large mixture-of-modality-experts model (MOME) that integrates multiparametric breast cancer MRI information within a unified structure, which shows reliable performance in a large cohort.

Abstract

Breast Magnetic Resonance Imaging (MRI) demonstrates the highest sensitivity for breast cancer detection among imaging modalities and is standard practice for high-risk women. Interpreting the multi-sequence MRI is time-consuming and prone to subjective variation. We develop a large mixture-of-modality-experts model (MOME) that integrates multiparametric MRI information within a unified structure, leveraging breast MRI scans from 5,205 female patients in China for model development and validation. MOME matches four senior radiologists' performance in identifying breast cancer and outperforms a junior radiologist. The model is able to reduce unnecessary biopsies in Breast Imaging-Reporting and Data System (BI-RADS) 4 patients, classify triple-negative breast cancer, and predict pathological complete response to neoadjuvant chemotherapy. MOME further supports inference with missing modalities and provides decision explanations by highlighting lesions and measuring modality contributions. To summarize, MOME exemplifies an accurate and robust multimodal model for noninvasive, personalized management of breast cancer patients via multiparametric MRI. Code is available at https://github.com/LLYXC/MOME/tree/main.

Figures (8)

• Figure 1: Overview of the multiparametric breast MRI-based study design. a. Data collection from three different hospitals, covering the population of the north, southeast, and southwest of China. b. The generation of multiparametric breast MRI, where T2-weighted MRI, Diffusion-weighted MRI, and DCE-MRI were mainly used in this study. c. MOME first takes multi-parametric MRI as input. Then, based on pre-trained foundation model parameters, mixture of sparse modality experts and soft modality experts are leveraged for unimodal feature extraction and multimodal information integration. d. MOME can be used for malignancy screening, molecular subtyping, and NACT response prediction, offering non-invasive personalized management for breast cancer patients. DS1 = dataset 1; DS2 = dataset 2; DS3 = dataset 3; MOME = mixture of modality experts; DCE = dynamic contrast-enhanced; DWI = diffusion-weighted Imaging; NACT = neoadjuvant chemotherapy. Multiple elements were created with BioRender.com (https://BioRender.com/w29a405).
• Figure 2: Discriminative malignancy detection performance of MOME. MOME achieved comparable MCC (a) and F1 (b) score performance to four experienced radiologists out of six readers, and significantly outperformed one junior radiologist, on the internal testing set 2 (n=200). MOME also showed high AUROC (c), pAUROC (d), and AURPC (e), on the internal testing set 2 (n=200). Moreover, MOME outperformed other unimodal or multimodal methods in both AUROC (f) and AUPRC (g), on the the combination of DS1 internal testing set 1 and DS2 (n=1042). In a-g, performance of all models were presented with CIs based on 1000-time bootstrap. In a, b, c, d, and e, error bars represent the 95% CIs. In f and g, the box shows the interquartile range (IQR) containing 50% of the data, with the bottom edge at Q1 (25th percentile), middle line at the median, and top edge at Q3 (75th percentile); the whiskers extend to the most extreme points within 1.5×IQR beyond the box edges, with points beyond the whiskers representing outliers. P-values were computed by comparing the performance of each method against that of MOME using bootstrapping. Rad = Radiologist; MOME = mixture of modality experts; MCC = Matthews's correlation coefficient; AUROC = area under the Receiver Operating Characteristic curve; AUPRC = area under the precision-recall curve; TPR = True Positive Rate; FPR = False Positive Rate; ***: p-value $\leq$ 0.001; **: p-value $\leq$ 0.01; *: p-value $<$ 0.05; ns: not significant. Source data are provided as a Source Data file.
• Figure 3: Malignancy diagnosis performance of MOME across different hospitals. The results correspond to the ROC curve (a), ROC curve with partial AUC (b), and precision-recall curve (c) on DS1 internal testing set; the ROC curve (d), ROC curve with partial AUC (e), and precision-recall curve (f) on DS2; and the ROC curve (g), ROC curve with partial AUC (h), and precision-recall curve (i) on DS3. The ROCs and precision-recall curves are drawn based on 1000-time bootstrap with 95% CI. Error bars represents the 95% CIs. AUROC = area under the Receiver Operating Characteristic curve; AUPRC = area under the precision-recall curve; pAUROC = partial area under the Receiver Operating Characteristic curve; TPR = True Positive Rate; FPR = False Positive Rate. Source data are provided as a Source Data file.
• Figure 4: Malignancy diagnosis performance of MOME on key subgroups on the combination of DS1 test set 1 and DS2. AUROC, AUPRC, sensitivity, and specificity are reported for each subgroup, from left to right. Red and green bars at the right represent the number of malignant and benign cases for each subgroup. All metrics are presented with 95% CI based on 1000-time bootstrap. AUROC = area under the Receiver Operating Characteristic curve; AUPRC = area under the precision-recall curve; BI-RADS = Breast Imaging-Reporting and Data System; BPE = Background Parenchymal Enhancement. DS1 = dataset 1; DS2 = dataset 2. Source data are provided as a Source Data file.
• Figure 5: Decision Interpretation of MOME. The illustrations correspond to DCE subtraction 3D visualization overlaid with saliencies in red computed by integrated gradient (a,b,g,h), the zoomed-in axial view of DCE subtraction, DWI, and T2WI (c,d,i,j), the local Shapley value (e,f,k,l), and global Shapley value of the DS1 internal testing set (m) and DS2 (n). Four typical cases of a BI-RADS 5 patient with a malignant lesion (a,c,e), a BI-RADS 4 patient with a benign lesion (b,d,f) from DS1 internal testing set, and a BI-RADS 5 patient with a malignant lesion (g,i,k), a BI-RADS 4 patient with a benign lesion (h,j,i) from DS2 are shown. BI-RADS = Breast Imaging-Reporting and Data System; DCE = dynamic contrast-enhanced; DWI = Diffusion-weighted Imaging; T2WI = T2-weighted imaging. Source data are provided as a Source Data file.