A Monte Carlo Framework for Calibrated Uncertainty Estimation in Sequence Prediction

TL;DR

This paper proposes a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence, and proposes a time-dependent regularization method, which is shown to produce calibrated predictions.

Abstract

Probabilistic prediction of sequences from images and other high-dimensional data is a key challenge, particularly in risk-sensitive applications. In these settings, it is often desirable to quantify the uncertainty associated with the prediction (instead of just determining the most likely sequence, as in language modeling). In this paper, we propose a Monte Carlo framework to estimate probabilities and confidence intervals associated with the distribution of a discrete sequence. Our framework uses a Monte Carlo simulator, implemented as an autoregressively trained neural network, to sample sequences conditioned on an image input. We then use these samples to estimate the probabilities and confidence intervals. Experiments on synthetic and real data show that the framework produces accurate discriminative predictions, but can suffer from miscalibration. In order to address this shortcoming, we propose a time-dependent regularization method, which is shown to produce calibrated predictions.

Figures (13)

• Figure 1: Sequence prediction with uncertainty estimation. The proposed framework enables estimation of marginal probabilities, conditional probabilities, and time-to-event confidence intervals associated with a sequence given an input image. We consider sequential decision-making tasks where the input image is a screenshot from an Atari video game and the sequence to predict is the sequence of actions taken by a user (top), and we consider a synthetic-data forecasting problem where the input image is the face of a person and the sequence to predict is the evolution of their health status (bottom).
• Figure 2: Monte Carlo framework for uncertainty estimation in sequence prediction. (a) A neural network simulator is trained to autoregressively predict the conditional distribution of each entry in a sequence given an input image and the preceding states. (b) The simulator is used to generate multiple sample sequences, by iteratively sampling from the estimated conditional distribution. (c) The Monte Carlo method is applied to estimate marginal probabilities, conditional probabilities, and time-to-event confidence intervals from the samples.
• Figure 3: Entry-wise calibration error and reliability diagrams for marginal probability estimation. The large graphs plot the entry-level ECE of the proposed foCus framework for estimation of marginal probabilities (see \ref{['sec:problem_statement']}) without regularization (black line, see \ref{['sec:pathology']}) and time-dependent regularization (red line, see \ref{['sec:regularization']}). Unregularized foCus produces miscalibrated estimates, particularly in the earlier entries, which are dramatically improved by time-dependent regularization for all datasets. The small graph show reliability diagrams for some of the steps, which confirm the improvement in calibration. Additional reliability diagrams and results for constant regularization are shown in \ref{['app:additional_results_marginal']}.
• Figure 4: Confidence intervals for time-to-event prediction and coverage probability. The upper panel shows heatmaps of the length of 0.9 confidence intervals for time-to-event prediction using the proposed foCus framework without and with time-dependent regularization. The histograms below show the frequency of intervals containing the true times, as a function of the true time. Unregularized foCus produces short intervals with poor coverage, whereas regularization yields intervals that tend to be larger when the ground-truth times are larger, and are much better calibrated. The plots correspond to the FaceMed (left) and Seaquest (right) datasets. \ref{['app:additional_results_ci']} shows analogous plots for the remaining datasets.
• Figure 5: Markov process used to simulate health-status transitions in FaceMed.