Exploiting Causality Signals in Medical Images: A Pilot Study with Empirical Results

TL;DR

This work introduces a causality-driven augmentation to CNNs for medical image classification by extracting causality factors from feature-map interactions to weight maps via a Mulcat module. Causality signals are captured as asymmetric relationships between feature maps using $P(F^i|F^j)$ and $P(F^j|F^i)$ estimated with Max or Lehmer means, then used to produce a causality-driven feature representation. Across BreakHis and PI-CAI datasets, Mulcat variants consistently improve accuracy and, in qualitative CAM analyses, focus attention on diagnostically relevant regions, with only a small memory overhead and compatibility with existing attention modules like BAM. The approach also demonstrates potential in few-shot learning and enhances explainability, suggesting a practical, data-driven path to integrating causal reasoning into medical image analysis.

Abstract

We present a novel technique to discover and exploit weak causal signals directly from images via neural networks for classification purposes. This way, we model how the presence of a feature in one part of the image affects the appearance of another feature in a different part of the image. Our method consists of a convolutional neural network backbone and a causality-factors extractor module, which computes weights to enhance each feature map according to its causal influence in the scene. We develop different architecture variants and empirically evaluate all the models on two public datasets of prostate MRI images and breast histopathology slides for cancer diagnosis. We study the effectiveness of our module both in fully-supervised and few-shot learning, we assess its addition to existing attention-based solutions, we conduct ablation studies, and investigate the explainability of our models via class activation maps. Our findings show that our lightweight block extracts meaningful information and improves the overall classification, together with producing more robust predictions that focus on relevant parts of the image. That is crucial in medical imaging, where accurate and reliable classifications are essential for effective diagnosis and treatment planning.

Figures (8)

• Figure 1: Zoomed-in visualizations of a sample causality map computed with Eq. \ref{['eq:causality_method_max']} on $512$ feature maps extracted from an input image. (a) $512 \times 512$ original causality map; (b) $256 \times 256$ zoom-in of (a); (c) $64 \times 64$ zoom-in of (b), where dashed circles indicate exemplar elements and their corresponding elements opposite the main diagonal, representing conditional asymmetries of the type $P(F^i|F^j) \neq P(F^j|F^i)$. We can see that, for instance, $P(F^{25}|F^{13}) > P(F^{13}|F^{25})$, that is $F^{25} \rightarrow F^{13}$, and $P(F^{45}|F^{40}) > P(F^{40}|F^{45})$, that is $F^{45} \rightarrow F^{40}$.
• Figure 2: Overview of the different settings investigated in this work: Baseline, Cat, and the proposed Mulcat. Assuming $k=6$ feature maps as an example, the tensor F of feature maps that are obtained from a CNN just before the classifier can be either flattened and used as they are (Option baseline) or can be leveraged to compute the causality map C via the Max or Lehmer method. Once obtained, C can be flattened as well and concatenated to the feature maps (Option Cat) or fed to our proposed causality factors extractor (see Figure \ref{['fig:caufacextractor']}) to implement the Option Mulcat. The latter produces a vector of causality factors that weighs the feature maps obtaining a causality-driven version of them, which is then concatenated to the original ones and fed to the classifier. Weighing mode m and causality direction d are two external signals used to tune the functioning of the system. This image is best seen in color.
• Figure 3: The internals of the proposed causality factors extractor block of Figure \ref{['fig:overview']} given an example causality map. Cyan squares in the causality map indicate whether the probability value of one element is greater than its element opposite the main diagonal. The causes box shows how the causality map is processed row-wise for each feature map: the number of times that feature is a cause of another feature is registered. Similarly, the effects box shows how the causality map is processed column-wise for each feature map. Before being summed element-wise, those two vectors are either passed as they are or the sign of their elements is reversed according to the causality direction d. The obtained vector is rectified and then returned as it is or passed through boolean filtering depending on the weighing mode m. This image is best seen in color.
• Figure 4: Some benign and malignant samples from the utilized BreakHis dataset. This image is best seen in color.
• Figure 5: Some no-tumour and cancerous samples from the utilized PI-CAI dataset.