Convolutional Neural Networks Rarely Learn Shape for Semantic Segmentation
Yixin Zhang, Maciej A. Mazurowski
TL;DR
This work presents a systematic study of whether CNNs learn shape for semantic segmentation, introducing the Shape Bias Index ($SBI$) to quantify reliance on shape versus non-shape cues. Using a probing pipeline with datasets $D_{val}$, $D_{rm}$, $D_{aff}$, and $D_{shuf}$ and both synthetic and real data, it shows CNNs rarely rely on shape under typical conditions and learn shape only when shape is the sole discriminative cue, aided by sufficiently large receptive fields. It further demonstrates that specific augmentations, notably Color Jitter and Neural Style Transfer (NST), can promote shape learning and improve generalization to Noisy and Out-of-Distribution (OOD) data, while other augmentations that modify shape may reduce SBI. These findings offer practical guidance on when and how to encourage shape information in segmentation models and highlight the tradeoffs with receptive field size and data augmentation in domain-shift settings.
Abstract
Shape learning, or the ability to leverage shape information, could be a desirable property of convolutional neural networks (CNNs) when target objects have specific shapes. While some research on the topic is emerging, there is no systematic study to conclusively determine whether and under what circumstances CNNs learn shape. Here, we present such a study in the context of segmentation networks where shapes are particularly important. We define shape and propose a new behavioral metric to measure the extent to which a CNN utilizes shape information. We then execute a set of experiments with synthetic and real-world data to progressively uncover under which circumstances CNNs learn shape and what can be done to encourage such behavior. We conclude that (i) CNNs do not learn shape in typical settings but rather rely on other features available to identify the objects of interest, (ii) CNNs can learn shape, but only if the shape is the only feature available to identify the object, (iii) sufficiently large receptive field size relative to the size of target objects is necessary for shape learning; (iv) a limited set of augmentations can encourage shape learning; (v) learning shape is indeed useful in the presence of out-of-distribution data.