Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Utilising Deep Learning to Elicit Expert Uncertainty

Julia R. Falconer, Eibe Frank, Devon L. L. Polaschek, Chaitanya Joshi

TL;DR

This work tackles the problem of eliciting expert uncertainty from complex, non-tabular data by extending the prior-elicitation framework to deep learning, enabling models to leverage the information used by experts in real decisions. It details probabilistic deep-learning approaches, notably Bayesian neural networks and MC-Dropout, to produce predictive distributions that reflect uncertainty. The authors demonstrate the method with a colon cancer risk assessment using histopathology images labeled by seven pathologists, training a ResNet-18 with dropout and fitting Beta distributions to samples to reflect expert uncertainty. The results show the approach can generate calibrated, conservative distributions that align with levels of expert agreement, suggesting practical utility for Bayesian inference, decision-making, and risk analysis in healthcare and beyond.

Abstract

Recent work [ 14 ] has introduced a method for prior elicitation that utilizes records of expert decisions to infer a prior distribution. While this method provides a promising approach to eliciting expert uncertainty, it has only been demonstrated using tabular data, which may not entirely represent the information used by experts to make decisions. In this paper, we demonstrate how analysts can adopt a deep learning approach to utilize the method proposed in [14 ] with the actual information experts use. We provide an overview of deep learning models that can effectively model expert decision-making to elicit distributions that capture expert uncertainty and present an example examining the risk of colon cancer to show in detail how these models can be used.

Utilising Deep Learning to Elicit Expert Uncertainty

TL;DR

This work tackles the problem of eliciting expert uncertainty from complex, non-tabular data by extending the prior-elicitation framework to deep learning, enabling models to leverage the information used by experts in real decisions. It details probabilistic deep-learning approaches, notably Bayesian neural networks and MC-Dropout, to produce predictive distributions that reflect uncertainty. The authors demonstrate the method with a colon cancer risk assessment using histopathology images labeled by seven pathologists, training a ResNet-18 with dropout and fitting Beta distributions to samples to reflect expert uncertainty. The results show the approach can generate calibrated, conservative distributions that align with levels of expert agreement, suggesting practical utility for Bayesian inference, decision-making, and risk analysis in healthcare and beyond.

Abstract

Recent work [ 14 ] has introduced a method for prior elicitation that utilizes records of expert decisions to infer a prior distribution. While this method provides a promising approach to eliciting expert uncertainty, it has only been demonstrated using tabular data, which may not entirely represent the information used by experts to make decisions. In this paper, we demonstrate how analysts can adopt a deep learning approach to utilize the method proposed in [14 ] with the actual information experts use. We provide an overview of deep learning models that can effectively model expert decision-making to elicit distributions that capture expert uncertainty and present an example examining the risk of colon cancer to show in detail how these models can be used.
Paper Structure (10 sections, 2 equations, 7 figures, 3 tables)

This paper contains 10 sections, 2 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Uncertainty
  3. Deep Learning
  4. Probabilistic Deep Learning
  5. Diagnosis Example
  6. Data
  7. Model
  8. Model Performance
  9. Elicited Distribution
  10. Conclusions and Future Research

Figures (7)

  • Figure 1: Multiple passes through a NN with dropout layers for a single input will produce different results.
  • Figure 2: Basic Model Structure and Incoporation of Dropout Layers
  • Figure 3: Entropy Plots for Cancer Diagnosis Example
  • Figure 4: Model Diagnostic Plots for Cancer Diagnosis Example
  • Figure 5: Entropy Plots of the Elicited Distributions of all Test Points. Split by Agreement Levels.
  • ...and 2 more figures