Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Deep Learning Model Calibration for Cardiac Applications using Deterministic Uncertainty Networks and Uncertainty-aware Training

Tareen Dawood, Bram Ruijsink, Reza Razavi, Andrew P. King, Esther Puyol-Antón

TL;DR

This work tackles the calibration gap in deep learning for high-risk cardiac imaging tasks by systematically evaluating three deterministic uncertainty models (DUMs) and two uncertainty-aware training strategies. It compares ENN, DDU, and LDU architectures, and combines them with AvUC and MMCE losses, across two clinically relevant datasets: PC-CMR artefact detection and ACDC cardiac disease diagnosis. The study finds that DUMs generally yield stronger calibration improvements, with DDU and LDU often providing the best balance of accuracy and calibration, and that incorporating uncertainty-aware losses can yield additional benefits, especially when paired with MMCE. A key contribution is the demonstration that a novel deterministic uncertainty-aware training approach can further enhance calibration, supporting more trustworthy AI-assisted decision making in cardiovascular imaging.

Abstract

Improving calibration performance in deep learning (DL) classification models is important when planning the use of DL in a decision-support setting. In such a scenario, a confident wrong prediction could lead to a lack of trust and/or harm in a high-risk application. We evaluate the impact on accuracy and calibration of two types of approach that aim to improve DL classification model calibration: deterministic uncertainty methods (DUM) and uncertainty-aware training. Specifically, we test the performance of three DUMs and two uncertainty-aware training approaches as well as their combinations. To evaluate their utility, we use two realistic clinical applications from the field of cardiac imaging: artefact detection from phase contrast cardiac magnetic resonance (CMR) and disease diagnosis from the public ACDC CMR dataset. Our results indicate that both DUMs and uncertainty-aware training can improve both accuracy and calibration in both of our applications, with DUMs generally offering the best improvements. We also investigate the combination of the two approaches, resulting in a novel deterministic uncertainty-aware training approach. This provides further improvements for some combinations of DUMs and uncertainty-aware training approaches.

Improving Deep Learning Model Calibration for Cardiac Applications using Deterministic Uncertainty Networks and Uncertainty-aware Training

TL;DR

This work tackles the calibration gap in deep learning for high-risk cardiac imaging tasks by systematically evaluating three deterministic uncertainty models (DUMs) and two uncertainty-aware training strategies. It compares ENN, DDU, and LDU architectures, and combines them with AvUC and MMCE losses, across two clinically relevant datasets: PC-CMR artefact detection and ACDC cardiac disease diagnosis. The study finds that DUMs generally yield stronger calibration improvements, with DDU and LDU often providing the best balance of accuracy and calibration, and that incorporating uncertainty-aware losses can yield additional benefits, especially when paired with MMCE. A key contribution is the demonstration that a novel deterministic uncertainty-aware training approach can further enhance calibration, supporting more trustworthy AI-assisted decision making in cardiovascular imaging.

Abstract

Improving calibration performance in deep learning (DL) classification models is important when planning the use of DL in a decision-support setting. In such a scenario, a confident wrong prediction could lead to a lack of trust and/or harm in a high-risk application. We evaluate the impact on accuracy and calibration of two types of approach that aim to improve DL classification model calibration: deterministic uncertainty methods (DUM) and uncertainty-aware training. Specifically, we test the performance of three DUMs and two uncertainty-aware training approaches as well as their combinations. To evaluate their utility, we use two realistic clinical applications from the field of cardiac imaging: artefact detection from phase contrast cardiac magnetic resonance (CMR) and disease diagnosis from the public ACDC CMR dataset. Our results indicate that both DUMs and uncertainty-aware training can improve both accuracy and calibration in both of our applications, with DUMs generally offering the best improvements. We also investigate the combination of the two approaches, resulting in a novel deterministic uncertainty-aware training approach. This provides further improvements for some combinations of DUMs and uncertainty-aware training approaches.
Paper Structure (38 sections, 8 equations, 2 figures, 4 tables)

This paper contains 38 sections, 8 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Model Calibration
  4. Improving Calibration
  5. Post-hoc Methods
  6. Uncertainty Aware Training
  7. Improving Uncertainty Estimates
  8. Dirichlet Distributions and Evidential Neural Networks:
  9. Deep Deterministic Uncertainty Networks:
  10. Latent Discriminant Deterministic Uncertainty Networks:
  11. Contributions
  12. Methods
  13. Baseline Model
  14. Deterministic Uncertainty Models
  15. Evidential Neural Network
  16. ...and 23 more sections

Figures (2)

  • Figure 1: Illustration of the baseline classification model with a 3D ResNet50 architecture. Images in (a) represent the phase contrast cardiac magnetic resonance (CMR) data and (b) the ACDC CMR dataset respectively.
  • Figure 2: Adaptations to the baseline classification model (Figure \ref{['fig:base_model']}) to implement the three deterministic uncertainty methods. (a) Evidential neural network, (b) Deep deterministic uncertainty network, (c) Latent discriminant deterministic uncertainty network. See text for further details.