Neighbor-Aware Calibration of Segmentation Networks with Penalty-Based Constraints

TL;DR

This paper tackles miscalibration in semantic segmentation, especially for medical imaging, where reliable uncertainty estimates are critical. It analyzes Spatially Varying Label Smoothing (SVLS) and reveals that SVLS imposes an implicit surrounding-class constraint on soft labels without a tunable balance to the main objective. The authors propose Neighbor Aware CaLibration (NACL), an approach that enforces equality constraints on logits via a differentiable penalty, enabling explicit control over both the prior and its influence during training. Across six diverse segmentation benchmarks and multiple backbones, NACL delivers superior calibration with preserved or improved discriminative performance, showing model-agnostic applicability and robustness to hyperparameters and data availability. This method holds practical potential for safer, more reliable decision-making in medical imaging and other dense-prediction tasks.

Abstract

Ensuring reliable confidence scores from deep neural networks is of paramount significance in critical decision-making systems, particularly in real-world domains such as healthcare. Recent literature on calibrating deep segmentation networks has resulted in substantial progress. Nevertheless, these approaches are strongly inspired by the advancements in classification tasks, and thus their uncertainty is usually modeled by leveraging the information of individual pixels, disregarding the local structure of the object of interest. Indeed, only the recent Spatially Varying Label Smoothing (SVLS) approach considers pixel spatial relationships across classes, by softening the pixel label assignments with a discrete spatial Gaussian kernel. In this work, we first present a constrained optimization perspective of SVLS and demonstrate that it enforces an implicit constraint on soft class proportions of surrounding pixels. Furthermore, our analysis shows that SVLS lacks a mechanism to balance the contribution of the constraint with the primary objective, potentially hindering the optimization process. Based on these observations, we propose NACL (Neighbor Aware CaLibration), a principled and simple solution based on equality constraints on the logit values, which enables to control explicitly both the enforced constraint and the weight of the penalty, offering more flexibility. Comprehensive experiments on a wide variety of well-known segmentation benchmarks demonstrate the superior calibration performance of the proposed approach, without affecting its discriminative power. Furthermore, ablation studies empirically show the model agnostic nature of our approach, which can be used to train a wide span of deep segmentation networks.

Figures (11)

• Figure 1: Compromise between calibration and discriminative performance. For each dataset, we show the discriminative (DSC) and calibration (ECE) results obtained by each method. We expect a well-calibrated model to achieve simultaneously large DSC (in blue) and small ECE (in brown) values.
• Figure 2: Ranking global and per-metric of the different methods based on the sum-rank and mean of case-specific approach.
• Figure 3: Impact of applying the penalty over softmax (cross) vs logits (circle) predictions across the different datasets.
• Figure 4: Distribution of logit predictions provided by a model trained with CE+DSC, LS, MbLS, SVLS and our approach (from left to right) on FLARE (top) and ACDC (bottom).
• Figure 5: Histogram of global logit distribution over epochs obtained by the different approaches.