Loss Functions in the Era of Semantic Segmentation: A Survey and Outlook

TL;DR

The paper surveys 25 loss functions for semantic segmentation, organized into Pixel-Level, Region-Level, Boundary-Level, and Combination categories, and provides a unified notation and taxonomy. It conducts controlled experiments on biomedical (Synapse) and natural (Cityscapes) datasets using CNN and Transformer-based models to compare loss performance, offering both quantitative and qualitative insights. The authors highlight hyperparameter sensitivity, practical guidelines, and open challenges such as label uncertainty, robustness to noisy annotations, and adaptation to foundation-model based pipelines. Overall, the work aims to guide practitioners in selecting and combining loss functions tailored to data characteristics and model architectures to advance segmentation performance in both medical and natural imaging domains.

Abstract

Semantic image segmentation, the process of classifying each pixel in an image into a particular class, plays an important role in many visual understanding systems. As the predominant criterion for evaluating the performance of statistical models, loss functions are crucial for shaping the development of deep learning-based segmentation algorithms and improving their overall performance. To aid researchers in identifying the optimal loss function for their particular application, this survey provides a comprehensive and unified review of $25$ loss functions utilized in image segmentation. We provide a novel taxonomy and thorough review of how these loss functions are customized and leveraged in image segmentation, with a systematic categorization emphasizing their significant features and applications. Furthermore, to evaluate the efficacy of these methods in real-world scenarios, we propose unbiased evaluations of some distinct and renowned loss functions on established medical and natural image datasets. We conclude this review by identifying current challenges and unveiling future research opportunities. Finally, we have compiled the reviewed studies that have open-source implementations on our GitHub page.

Figures (8)

• Figure 1: The taxonomy subsections delineate four distinct groups: (1) Region-Level, (2) Boundary-Level, (3) Pixel-Level, and (4) Combination.
• Figure 2: The qualitative comparison of segmentation results on the ACDC dataset from sun2023boundary. (Row 1&2: TransUNet, and Row 3&4: UNet)
• Figure 3: Impact of different loss functions on the segmentation of large (left plot) and small (right plot) objects. asgari2021deep.
• Figure 4: Loss function plots trained on the Synapse dataset for the UNet network on the left and TransUNet network on the right for the selected loss functions, i.e. Dice Loss, Focal Tversky Loss, Focal Loss, Jaccard Loss Lovász-Softmax Loss and Tversky. The loss values are averaged over all iterations in one epoch.
• Figure 5: Loss function plots trained on the Cityscapes dataset for the UNet network on the left and TransUNet network on the right for the selected loss functions, i.e. Dice Loss, Focal Tversky Loss, Focal Loss, Jaccard Loss, Lovász-Softmax Loss and Tversky. The loss values are averaged over all iterations in one epoch.