Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibrating Segmentation Networks with Margin-based Label Smoothing

Balamurali Murugesan, Bingyuan Liu, Adrian Galdran, Ismail Ben Ayed, Jose Dolz

TL;DR

This work provides a unifying constrained-optimization perspective of current state-of-the-art calibration losses of deep neural networks, and proposes a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.

Abstract

Despite the undeniable progress in visual recognition tasks fueled by deep neural networks, there exists recent evidence showing that these models are poorly calibrated, resulting in over-confident predictions. The standard practices of minimizing the cross entropy loss during training promote the predicted softmax probabilities to match the one-hot label assignments. Nevertheless, this yields a pre-softmax activation of the correct class that is significantly larger than the remaining activations, which exacerbates the miscalibration problem. Recent observations from the classification literature suggest that loss functions that embed implicit or explicit maximization of the entropy of predictions yield state-of-the-art calibration performances. Despite these findings, the impact of these losses in the relevant task of calibrating medical image segmentation networks remains unexplored. In this work, we provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. Specifically, these losses could be viewed as approximations of a linear penalty (or a Lagrangian term) imposing equality constraints on logit distances. This points to an important limitation of such underlying equality constraints, whose ensuing gradients constantly push towards a non-informative solution, which might prevent from reaching the best compromise between the discriminative performance and calibration of the model during gradient-based optimization. Following our observations, we propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances. Comprehensive experiments on a variety of public medical image segmentation benchmarks demonstrate that our method sets novel state-of-the-art results on these tasks in terms of network calibration, whereas the discriminative performance is also improved.

Calibrating Segmentation Networks with Margin-based Label Smoothing

TL;DR

This work provides a unifying constrained-optimization perspective of current state-of-the-art calibration losses of deep neural networks, and proposes a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances.

Abstract

Despite the undeniable progress in visual recognition tasks fueled by deep neural networks, there exists recent evidence showing that these models are poorly calibrated, resulting in over-confident predictions. The standard practices of minimizing the cross entropy loss during training promote the predicted softmax probabilities to match the one-hot label assignments. Nevertheless, this yields a pre-softmax activation of the correct class that is significantly larger than the remaining activations, which exacerbates the miscalibration problem. Recent observations from the classification literature suggest that loss functions that embed implicit or explicit maximization of the entropy of predictions yield state-of-the-art calibration performances. Despite these findings, the impact of these losses in the relevant task of calibrating medical image segmentation networks remains unexplored. In this work, we provide a unifying constrained-optimization perspective of current state-of-the-art calibration losses. Specifically, these losses could be viewed as approximations of a linear penalty (or a Lagrangian term) imposing equality constraints on logit distances. This points to an important limitation of such underlying equality constraints, whose ensuing gradients constantly push towards a non-informative solution, which might prevent from reaching the best compromise between the discriminative performance and calibration of the model during gradient-based optimization. Following our observations, we propose a simple and flexible generalization based on inequality constraints, which imposes a controllable margin on logit distances. Comprehensive experiments on a variety of public medical image segmentation benchmarks demonstrate that our method sets novel state-of-the-art results on these tasks in terms of network calibration, whereas the discriminative performance is also improved.
Paper Structure (30 sections, 1 theorem, 13 equations, 9 figures, 4 tables)

This paper contains 30 sections, 1 theorem, 13 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Calibration in medical image segmentation
  4. Preliminaries
  5. A constrained-optimization perspective of calibration
  6. Definition of logit distances
  7. Penalty functions in constrained optimization
  8. Margin-based Label Smoothing (MbLS)
  9. Experiments
  10. Experimental Setting
  11. Datasets
  12. Automated Cardiac Diagnosis Challenge (ACDC) bernard2018deep
  13. Brain Tumor Segmentation (BRATS) 2019 Challenge
  14. MRBrainS18 mrbrains_dataset
  15. Fast and Low GPU memory Abdominal oRgan sEgmentation (FLARE) Challenge AbdomenCT-1K
  16. ...and 15 more sections

Key Result

Proposition 1

A linear penalty (or a Lagrangian term) for constraint $\mathbf{d} (\mathbf{l}) = {\mathbf 0}$ is bounded from above and below by ${\cal D}_\text{KL}\left({\mathbf u} || \mathbf{s} \right )$, up to additive constants: where $\stackrel{\mathclap{\normalfont\hbox{c}}}{\leq}$ stands for inequality up to an additive constant.

Figures (9)

  • Figure 1: Illustration of the linear (left) and margin-based (right) penalties for imposing logit-distance constraints, along with the corresponding derivatives. Note that while the derivative of the linear penalty for constraint $\mathbf{d}(\mathbf{l})=\mathbf{0}$ constantly pushes towards the trivial solution $s_k=\frac{1}{K}\forall K$ (i.e., LS, FL and EPC), the derivative of the proposed model only pushes towards zero those logits above the given margin.
  • Figure 2: Compromise between calibration and discriminative performance. For each dataset, we show the discriminative (DICE) and calibration (ECE) results for each method. In order to get the best performance, we expect a model to achieve large DSC (in green) and small ECE (in blue) values.
  • Figure 3: Ranking (global and per-metric) of the different methods based on the sum-rank and mean of case-specific approach.
  • Figure 4: Adopting the proposed term during training substantially reduces the logit distances, producing less overconfident predictions. These plots depict the average predicted logit distributions for each target class --based on the ground truth-- on ACDC (top) and FLARE (bottom) datasets when the model is trained with CE (left) and the proposed loss (right).
  • Figure 5: Robustness to distributional drift on PROMISE (left) and MRBrainS (right) datasets. Note that larger circles represent lower sigma values for the Gaussian noise corruptions.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Proposition 1