Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Conformal confidence sets for biomedical image segmentation

Samuel Davenport

TL;DR

It is shown that learning appropriate score transformations on a learning dataset before performing calibration is crucial for optimizing performance and using distance transformed scores to obtain outer confidence sets and the original scores for inner confidence sets enables tight bounds on tumor location whilst controlling the false coverage rate.

Abstract

We develop confidence sets which provide spatial uncertainty guarantees for the output of a black-box machine learning model designed for image segmentation. To do so we adapt conformal inference to the imaging setting, obtaining thresholds on a calibration dataset based on the distribution of the maximum of the transformed logit scores within and outside of the ground truth masks. We prove that these confidence sets, when applied to new predictions of the model, are guaranteed to contain the true unknown segmented mask with desired probability. We show that learning appropriate score transformations on a learning dataset before performing calibration is crucial for optimizing performance. We illustrate and validate our approach on a polpys tumor dataset. To do so we obtain the logit scores from a deep neural network trained for polpys segmentation and show that using distance transformed scores to obtain outer confidence sets and the original scores for inner confidence sets enables tight bounds on tumor location whilst controlling the false coverage rate.

Conformal confidence sets for biomedical image segmentation

TL;DR

It is shown that learning appropriate score transformations on a learning dataset before performing calibration is crucial for optimizing performance and using distance transformed scores to obtain outer confidence sets and the original scores for inner confidence sets enables tight bounds on tumor location whilst controlling the false coverage rate.

Abstract

We develop confidence sets which provide spatial uncertainty guarantees for the output of a black-box machine learning model designed for image segmentation. To do so we adapt conformal inference to the imaging setting, obtaining thresholds on a calibration dataset based on the distribution of the maximum of the transformed logit scores within and outside of the ground truth masks. We prove that these confidence sets, when applied to new predictions of the model, are guaranteed to contain the true unknown segmented mask with desired probability. We show that learning appropriate score transformations on a learning dataset before performing calibration is crucial for optimizing performance. We illustrate and validate our approach on a polpys tumor dataset. To do so we obtain the logit scores from a deep neural network trained for polpys segmentation and show that using distance transformed scores to obtain outer confidence sets and the original scores for inner confidence sets enables tight bounds on tumor location whilst controlling the false coverage rate.
Paper Structure (25 sections, 8 theorems, 29 equations, 15 figures)

This paper contains 25 sections, 8 theorems, 29 equations, 15 figures.

Table of Contents

  1. Introduction
  2. Theory
  3. Set up
  4. Marginal confidence sets
  5. Joint confidence sets
  6. Optimizing score transformations
  7. Constructing confidence sets from bounding boxes
  8. Application to Polpys tumor segmentation
  9. Choosing a score transformation
  10. Illustrating the performance of conformal confidence sets
  11. Measuring the coverge rate
  12. Comparing the efficiency of the bounds
  13. Discussion
  14. Appendix
  15. Obtaining conformal confidence sets with increasing combination functions
  16. ...and 10 more sections

Key Result

Theorem 2.1

(Marginal inner set) Under Assumptions ass:ex and ass:indep, given $\alpha_1 \in (0,1)$, let and define $I(X) = \lbrace v \in \mathcal{V}: f_I(s(X), v) >\lambda_I(\alpha_1) \rbrace$. Then,

Figures (15)

  • Figure 1: Histograms of the distribution of the scores over the whole image within and outside the ground truth masks. Thresholds obtained for the marginal $90\%$ inner and outer confidence sets, obtained based on quantiles of the distribution of $(\tau_i)_{i = 1}^n$ and $(\gamma_i)_{i = 1}^n$, are displayed in red and blue.
  • Figure 2: Illustrating the performance of the different score transformations on the learning dataset. We display 2 example tumors and present the results of each in 8 panels. These panels are as follows. Bottom left: the original image of the polpys tumor. Top Left: an intensity plot of the scores obtained from PraNet with purple/yellow indicating areas of lower/higher assigned probability. For the remaining panels, 3 different score transformations are shown which from left to right are the original scores, distance transformed scores $d_\rho(\hat{M}(X), v)$ and bounding box scores (obtained using the combined bounding box score $b_M$ defined in Definition \ref{['dfn:BBS']}). In each of the panels on the top row a surface plot of the transformed PraNet scores is shown, along with the conformal thresholds which are used to obtain the marginal 90% inner and outer confidence sets. These thresholds are illustrated via red and blue planes respectively and are obtained over the learning dataset. The panels on the bottom row of each example show the corresponding conformal confidence sets. Here the inner set is shown in red, plotted over the ground truth mask of the polyps, shown in yellow, plotted over the outer set which is shown in blue. The outer set contains the ground truth mask which contains the inner set in all examples. From these figures we see that the original scores provide tight inner confidence sets and the distance transformed scores instead provide tight outer confidence sets. The conclusion from the learning dataset is therefore that it makes sense to combine these two score transformations.
  • Figure 3: Conformal confidence sets for the polyps data. For each set of polpys images the top row shows the original endoscopic images with visible polyps and the second row presents the marginal 90% confidence sets, with ground truth masks shown in yellow. The inner sets and outer sets are shown in red and blue, obtained using the identity and distance transforms respectively. The figure shows the benefits of combining different score transformations for the inner and outer sets and illustrates the method's effectiveness in accurately identifying polyp regions whilst providing informative spatial uncertainty bounds.
  • Figure 4: Coverage levels of the inner and outer sets averaged over 1000 validations for the original, distance transformed (DT) and bounding box (BB) scores.
  • Figure 5: Measuring the efficiency of the bound using the ratio of the diameter of the coverage set to the diameter of the true tumor mask. The closer the ratio is to one the better. Higher coverage rates lead to a lower efficiency. The original scores provide the most efficient inner sets and the distance transformed scores provide the most efficient outer sets.
  • ...and 10 more figures

Theorems & Definitions (16)

  • Theorem 2.1
  • proof
  • Theorem 2.2
  • proof
  • Remark 2.3
  • Remark 2.4
  • Corollary 2.5
  • Theorem 2.6
  • proof
  • Remark 2.7
  • ...and 6 more