Self-Supervised Learning for Image Segmentation: A Comprehensive Survey

TL;DR

This survey analyzes the rise of self-supervised learning (SSL) for image segmentation, focusing on how surrogate pretext tasks enable learning from unlabeled data and subsequent fine-tuning on segmentation tasks. It classifies pretext approaches into predictive, generative, and contrastive methods, and surveys their adaptations to semantic, instance, and panoptic segmentation across medical and urban domains. The work also catalogs commonly used datasets, compares representative SSL methods (e.g., CPC, SimCLR, MoCo, BYOL, SwAV, SimSiam, PGL), and discusses challenges and future directions such as domain adaptation, few-shot/zero-shot learning, and real-time segmentation. The paper aims to clarify the SSL segmentation landscape and guide researchers and practitioners in applying unlabeled data to dense visual prediction tasks.

Abstract

Supervised learning demands large amounts of precisely annotated data to achieve promising results. Such data curation is labor-intensive and imposes significant overhead regarding time and costs. Self-supervised learning (SSL) partially overcomes these limitations by exploiting vast amounts of unlabeled data and creating surrogate (pretext or proxy) tasks to learn useful representations without manual labeling. As a result, SSL has become a powerful machine learning (ML) paradigm for solving several practical downstream computer vision problems, such as classification, detection, and segmentation. Image segmentation is the cornerstone of many high-level visual perception applications, including medical imaging, intelligent transportation, agriculture, and surveillance. Although there is substantial research potential for developing advanced algorithms for SSL-based semantic segmentation, a comprehensive study of existing methodologies is essential to trace advances and guide emerging researchers. This survey thoroughly investigates over 150 recent image segmentation articles, particularly focusing on SSL. It provides a practical categorization of pretext tasks, downstream tasks, and commonly used benchmark datasets for image segmentation research. It concludes with key observations distilled from a large body of literature and offers future directions to make this research field more accessible and comprehensible for readers.

Figures (19)

• Figure 1: Three widely used image segmentation techniques.
• Figure 2: SSL-driven model development. (a) data preparation - $\mathcal{T}(\cdot)$ create synthetic samples $\mathbf{\tilde{X}}$ with pseudo labels $\mathbf{Y_p}$ from the unlabeled data $\mathbf{X}$ (b) self-supervised training - a model is trained using $\{\mathbf{\tilde{X}}, \mathbf{Y_p}\}$, (c) fine-tuning - the model is fine-tuned on a downstream task using a labeled dataset $\{\mathbf{X_s}, \mathbf{Y_s}\}$, and (d) performance rating - the final model is evaluated using test data of the target application. The unlabeled and labeled samples are adopted, respectively from Nilsback06 and zhou2017scene.
• Figure 3: For a given (a) Input, (b) instance segmentation: per-object mask and class label, (c) semantic segmentation, and (d) panoptic segmentation: per-pixel class + instance-level labels.
• Figure 4: Illustration of the Jigsaw puzzle. An input image is divided into titles ($\mathbf{\mathcal{X}}$) and then shuffled ($\tilde{\mathbf{\mathcal{X}}}$), where $\mathbf{\mathcal{Y}}$ is the pseudo label denoting the correct order of the titles. The model is trained to rearrange the shuffled titles to the correct order.
• Figure 5: Illustration of slice order prediction pretext task. The image inside the block is adopted from misra2016shuffle.