Soft-CAM: Making black box models self-explainable for high-stakes decisions

TL;DR

SoftCAM addresses the interpretability gap in CNNs for high-stakes medical imaging by replacing the fully connected head with a 1×1 convolutional class-evidence layer, producing explicit class evidence maps used for prediction. By applying ElasticNet regularization to these evidence maps, SoftCAM balances sparsity and completeness of explanations while maintaining competitive predictive performance. Across three diverse medical datasets, SoftCAM yields inherently faithful and localized explanations that outperform post-hoc CAM methods, enabling end-to-end self-explainability in common CNN backbones. The approach offers a practical, resource-efficient path toward trustworthy, self-explanatory deep learning in clinical decision-making and can be extended to other architectures and modalities.

Abstract

Convolutional neural networks (CNNs) are widely used for high-stakes applications like medicine, often surpassing human performance. However, most explanation methods rely on post-hoc attribution, approximating the decision-making process of already trained black-box models. These methods are often sensitive, unreliable, and fail to reflect true model reasoning, limiting their trustworthiness in critical applications. In this work, we introduce SoftCAM, a straightforward yet effective approach that makes standard CNN architectures inherently interpretable. By removing the global average pooling layer and replacing the fully connected classification layer with a convolution-based class evidence layer, SoftCAM preserves spatial information and produces explicit class activation maps that form the basis of the model's predictions. Evaluated on three medical datasets, SoftCAM maintains classification performance while significantly improving both the qualitative and quantitative explanation compared to existing post-hoc methods. Our results demonstrate that CNNs can be inherently interpretable without compromising performance, advancing the development of self-explainable deep learning for high-stakes decision-making.

Figures (18)

• Figure 1: Overview of softCAM for making black-box CNNs inherently interpretable. (a) Input image. (b) The CNN backbone consists of all layers before the global average pooling layer. (c) Feature map generated by the backbone. (d) Classifier module with $C$ convolutional kernels of size $1 \times 1$. (e) Self-explainable class activation maps $\mathbf{A}$, obtained from the classifier with ElasticNet penalty applied to it to enhance interpretability. (f) Final predictions are derived directly from the evidence maps via spatial average pooling followed by the softmax function. Class-specific evidence maps (g) are upsampled and overlaid on the input to visualize the model's decision-making process.
• Figure 2: Example explanations generated by different methods from ResNet-50. The first column shows disease images with reference annotations, indicated by green markers or bounding boxes. Each row, from top to bottom, corresponds to fundus, OCT, and Chest X-ray images, respectively. The next five columns present saliency maps generated by post-hoc explanation methods, gradient-free (ScoreCAM, LayerCAM) and gradient-based (GradCAM, Guided BP, Itgd Grad). The final two columns showcase our proposed inherently interpretable dense and sparse SoftCAM explanations.
• Figure 3: Quantitative evaluation of explanations generated by different methods. The first row shows the localization precision of the saliency maps on the Fundus and OCT datasets, evaluated against their respective ground truth. The second row presents the sensitivity analysis assessing the faithfulness of the generated explanations. Columns a,b show ResNet results, while c,d correspond to VGG. Higher precision means better localization; lower sensitivity implies more reliable explanations.
• Figure 4: Example of localization evaluation on the CXR dataset for pneumonia detection. The first row shows saliency maps generated by different methods from the ResNet model, and the second row from the VGG model. Ground-truth bounding boxes are overlaid on each map, with the top-right value indicating the activation precision, while the top-left value indicates the activation sensitivity.
• Figure 5: Examples of multi-class explanations using ResNet. For a severe DR example from the Kaggle dataset, the first row shows class-specific dense SoftCAM evidence map explanations, while the second presents explanations from the sparse SoftCAM.