Shortcut Learning in Medical Image Segmentation

TL;DR

It is demonstrated that clinical annotations such as calipers, and the combination of zero-padded convolutions and center-cropped training sets in the dataset can inadvertently serve as shortcuts, impacting segmentation accuracy.

Abstract

Shortcut learning is a phenomenon where machine learning models prioritize learning simple, potentially misleading cues from data that do not generalize well beyond the training set. While existing research primarily investigates this in the realm of image classification, this study extends the exploration of shortcut learning into medical image segmentation. We demonstrate that clinical annotations such as calipers, and the combination of zero-padded convolutions and center-cropped training sets in the dataset can inadvertently serve as shortcuts, impacting segmentation accuracy. We identify and evaluate the shortcut learning on two different but common medical image segmentation tasks. In addition, we suggest strategies to mitigate the influence of shortcut learning and improve the generalizability of the segmentation models. By uncovering the presence and implications of shortcuts in medical image segmentation, we provide insights and methodologies for evaluating and overcoming this pervasive challenge and call for attention in the community for shortcuts in segmentation. Our code is public at https://github.com/nina-weng/shortcut_skinseg .

Figures (6)

• Figure 1: Left: Shortcuts as yellow markers that affect ultrasound segmentation. The calipers were dilated for illustration. Right: Shortcuts that stem from the dataset construction, where lesions are center-cropped.
• Figure 2: Illustration of the dynamic variation of model segmentation prediction on the maternal cervix in a video. The similarity score between frames at time $t_0$ and the last frame $t_{end}$ is computed by dice score. The selected frame shows that the segmentation prediction is unstable when putting the calipers and texts.
• Figure 3: Sample lesion images from $\mathbf{ISIC_{CROP}}$ with ground truth (GT) segmentation and predicted segmentation by $M_{ori}$ and $M_{crop}$. We clearly see shortcut learning and its mitigation when training on $M_{ori}$ and $M_{crop}$, respectively.
• Figure 4: To emphasize the effect of shortcut learning, we separate the image into yellow bands of increasing distance from the center (a) and compute segmentation Dice scores within each region. Note how the Dice score of the shortcut model $M_{ori}$ declines with increasing distance from the center (b), while the mitigated model $M_{crop}$ maintains its segmentation ability across the image (c).
• Figure 5: Zero padding introduces a change in the features of segmented pixels whose receptive field contains padding -- i.e. those that are near the boundary. This affects their encoding in feature space.