Semantic segmentation with coarse annotations

TL;DR

The paper tackles semantic segmentation under coarse annotations, where boundary alignment is challenging due to unlabeled pixels near boundaries. It introduces a SLIC-based regularization that biases HCFCN-16's superpixels to align with color–positional priors, implemented via a pooling operator $q$ and a loss $L_{SLIC}$ that augments the standard cross-entropy loss with a scalar weight $\lambda$. Empirically, this regularization yields significant boundary recall improvements, especially on the difficult SUIM dataset, while incurring modest training-time overhead and maintaining competitive pixel accuracy on Cityscapes and PanNuke under coarse supervision. The method demonstrates that leveraging color-based superpixel structure can meaningfully enhance boundary fidelity when fine annotations are unavailable, enabling cheaper data collection for boundary-sensitive segmentation tasks.

Abstract

Semantic segmentation is the task of classifying each pixel in an image. Training a segmentation model achieves best results using annotated images, where each pixel is annotated with the corresponding class. When obtaining fine annotations is difficult or expensive, it may be possible to acquire coarse annotations, e.g. by roughly annotating pixels in an images leaving some pixels around the boundaries between classes unlabeled. Segmentation with coarse annotations is difficult, in particular when the objective is to optimize the alignment of boundaries between classes. This paper proposes a regularization method for models with an encoder-decoder architecture with superpixel based upsampling. It encourages the segmented pixels in the decoded image to be SLIC-superpixels, which are based on pixel color and position, independent of the segmentation annotation. The method is applied to FCN-16 fully convolutional network architecture and evaluated on the SUIM, Cityscapes, and PanNuke data sets. It is shown that the boundary recall improves significantly compared to state-of-the-art models when trained on coarse annotations.

Figures (8)

• Figure 1: A fine and a coarse annotation from the Cityscapes dataset
• Figure 2: Architecture proposed by Imp_integration with convolutional layers $g^{(l)}$ and downsampling layers $d^{(l)}$. Assignment matrix $A^{(l)}$ is obtained at the layer that downsamples $x^{(l)}$ to $s^{(l+1)}$. The low resolution output $y^{(L)}$ is upsampled by the assignment matrices $A$.
• Figure 3: Illustration of the function $q(x, A)$ defined by equation \ref{['pooling']}. The superpixel boundaries are highlighted in yellow. The feature map $x$ is downsampled using the the superpixels before upsampling with the same superpixels. For simplicity, each pixel is assigned to a single superpixel.
• Figure 4: Generating a coarse annotation from a fine annotation on the SUIM dataset. The fine annotation (a) is eroded by 12 pixels resulting in the eroded annotation (b). Finally the Douglas-pecker algorithm is used to approximate the eroded annotation with a polygon to obtain the coarse annotation (c).
• Figure 5: Example results on the Cityscapes (top), PanNuke (middle) the SUIM (bottom) test sets. All methods are trained on coarse annotations and fine-tuned on the training and validation set. The Cityscapes images are cropped to match the PanNuke and SUIM images. Examples are chosen to highlight the impact of regularization.