FPN-IAIA-BL: A Multi-Scale Interpretable Deep Learning Model for Classification of Mass Margins in Digital Mammography

TL;DR

The paper tackles the need for interpretable deep learning in radiology, focusing on mass-margin classification in digital mammography. It introduces FPN-IAIA-BL, a multi-scale, prototype-based architecture that extends IAIA-BL with a Feature Pyramid Network to learn interpretable prototypes across scales, using cosine similarity with focal pooling and a three-stage training scheme guided by a fine-annotation loss. A large Duke Health dataset with lesion and negative examples and a dedicated negative class supports training, with radiologist-informed coefficients shaping prototype activations. Empirically, FPN-IAIA-BL yields localized, scale-aware explanations and competitive interpretability, achieving an average AUROC of $0.88$ (circumscribed $0.88$, indistinct $0.87$, spiculated $0.86$; overall $0.91$), though IAIA-BL remains stronger in raw accuracy. The work contributes a general, scalable architecture for case-based explanations in computer vision that can be adapted to other high-stakes domains.

Abstract

Digital mammography is essential to breast cancer detection, and deep learning offers promising tools for faster and more accurate mammogram analysis. In radiology and other high-stakes environments, uninterpretable ("black box") deep learning models are unsuitable and there is a call in these fields to make interpretable models. Recent work in interpretable computer vision provides transparency to these formerly black boxes by utilizing prototypes for case-based explanations, achieving high accuracy in applications including mammography. However, these models struggle with precise feature localization, reasoning on large portions of an image when only a small part is relevant. This paper addresses this gap by proposing a novel multi-scale interpretable deep learning model for mammographic mass margin classification. Our contribution not only offers an interpretable model with reasoning aligned with radiologist practices, but also provides a general architecture for computer vision with user-configurable prototypes from coarse- to fine-grained prototypes.

Figures (6)

• Figure 1: Activation maps for FPN-IAIA-BL in comparison to IAIA-BL. FPN-IAIA-BL can learn human interpretable prototypes at any scale, including fine-grained details most salient to mass margin classification.
• Figure 2: FPN-IAIA-BL Architectre. The input image $\mathbf{x}$ passes through convolutional layers $f$ consisting of an FPN with a VGG-16 backbone, which creates an pyramid of feature maps $f(\mathbf{x})$. Each patch of each level of the feature pyramid (referred to as FPN level) is then compared to each prototype of the same FPN level using a cosine distance to produce an activation map. The activation map is then used to calculate an overall similarity score $s_j$ between the input image and the prototype for each prototype. Finally, a set of fully connected last layer produces logits $y_{\text{margin}}$ for each margin class.
• Figure 3: Case-based explanation generated by FPN-IAIA-BL. This circumscribed (circ.) lesion is correctly classified as circumscribed. a, Test images. b, Activation of prototype on test images. c, Most relevant part of prototype. d, Learned prototypical lesion. e, Prototype self-activation. f, Contribution to class score. This visualization format for this figure matches that of barnett2021case.
• Figure 4: FPN-IAIA-BL in comparison to other saliency methods (adapted from barnett2021case). We compare explanations from FPN-IAIA-BL with GradCAM Selvaraju_2017_ICCV, GradCAM++ chattopadhay2018grad, ProtoPNet PPNet, and IAIA-BL barnett2021case. GradCam and GradCAM++ are two popular saliency explanation methods, and ProtoPNet and IAIA-BL are case-based explanation methods. The explanations from FPN-IAIA-BL highlight the most important parts of the lesion margin.
• Figure 5: Learned prototypes at different FPN-levels. FPN-level 2 prototypes are more localized because they are learned from the base of the feature pyramid which is a finer-grained feature map while FPN-level 5 prototypes are learned from the top of the feature pyramid, a coarser-grained feature map.