Uncertainty quantification with approximate variational learning for wearable photoplethysmography prediction tasks

TL;DR

This paper tackles uncertainty quantification for wearable photoplethysmography (PPG) tasks by applying two scalable Bayesian-inspired methods—Monte Carlo Dropout (MCD) and Improved Variational Online Newton (IVON)—to atrial fibrillation (AF) classification and blood pressure (BP) regression from raw PPG time series. It systematically explores how model hyperparameters, notably stochasticity in sampling, affect predictive performance and the composition of epistemic and aleatoric uncertainty, using a comprehensive set of calibration metrics and adaptivity analyses. The authors introduce a framework for disentangling uncertainties and validating calibration at both global and per-class levels, highlighting challenges in truly separating uncertainty sources in practice. The study’s key finding is that hyperparameter tuning is essential for balancing accuracy and calibrated uncertainty, and that calibration quality can vary significantly across tasks, classes, and signal quality, underscoring the need for thorough evaluation before clinical deployment.

Abstract

Photoplethysmography (PPG) signals encode information about relative changes in blood volume that can be used to assess various aspects of cardiac health non-invasively, e.g.\ to detect atrial fibrillation (AF) or predict blood pressure (BP). Deep networks are well-equipped to handle the large quantities of data acquired from wearable measurement devices. However, they lack interpretability and are prone to overfitting, leaving considerable risk for poor performance on unseen data and misdiagnosis. Here, we describe the use of two scalable uncertainty quantification techniques: Monte Carlo Dropout and the recently proposed Improved Variational Online Newton. These techniques are used to assess the trustworthiness of models trained to perform AF classification and BP regression from raw PPG time series. We find that the choice of hyperparameters has a considerable effect on the predictive performance of the models and on the quality and composition of predicted uncertainties. E.g. the stochasticity of the model parameter sampling determines the proportion of the total uncertainty that is aleatoric, and has varying effects on predictive performance and calibration quality dependent on the chosen uncertainty quantification technique and the chosen expression of uncertainty. We find significant discrepancy in the quality of uncertainties over the predicted classes, emphasising the need for a thorough evaluation protocol that assesses local and adaptive calibration. This work suggests that the choice of hyperparameters must be carefully tuned to balance predictive performance and calibration quality, and that the optimal parameterisation may vary depending on the chosen expression of uncertainty.

Figures (5)

• Figure 1: Distributions of ground truth blood pressure (BP) values in the a) training set, b) validation set, and c) test set.
• Figure 2: Evaluation of uncertainty calibration for blood pressure (BP) regression models trained with Monte Carlo Dropout (MCD). a-c) show the ENCE and corresponding reliability diagrams for both systolic blood pressure (SBP) and diastolic blood pressure (DBP), where the RMV is the root mean variance of the uncertainties in a given bin. d-f) show the coverage-based reliability diagrams and corresponding coverage calibration error (CCE) values. g-l) show how the predicted uncertainty is distributed against the prediction error (truncated along the horizontal axis to improve visualisation) for SBP and DBP.
• Figure 3: Uncertainty disentanglement for blood pressure (BP) regression models trained with Monte Carlo Dropout (MCD). a-c) show scatterplots of the aleatoric vs total uncertainty estimated for each test example expressed as variances, along with histograms showing the distributions of each value for diastolic blood pressure (DBP). d-f) show the corresponding plots for systolic blood pressure (SBP). We find that increasing the dropout rate increases how much of the total uncertainty is composed of epistemic uncertainty.
• Figure 4: Evaluation of uncertainty calibration for Atrial Fibrillation (AF) classification models trained with Monte carlo Dropout (MCD). a-c) show the uncertainty calibration curves, d-f) show the calibration curves, and g-i) are scatterplots showing what proportion of the the total uncertainty for each test example that is aleatoric, along with the distribution of predicted uncertainties for each class. We find that the proportion of epistemic uncertainty is higher for models trained with higher dropout rates.
• Figure 5: Evaluation of uncertainty calibration for Atrial Fibrillation (AF) classification models trained with IVON. a-c) show the uncertainty calibration curves, d-f) show the calibration curves, and g-i) are scatterplots showing what proportion of the the total uncertainty for each test example that is aleatoric, along with the distribution of predicted uncertainties for each class. We find that the proportion of epistemic uncertainty is higher for models trained with lower $h_0$ values.