Evaluating infectious disease forecasts in a cost-loss situation

TL;DR

The paper addresses the mismatch between traditional forecast evaluation and decision-making needs by introducing a cost–loss based Value Score (VS) that quantifies the economic value of epidemic forecasts relative to a historical baseline. By applying VS to FluSight influenza peak forecasts, the authors show that several models provide added value for certain cost–loss ratios, even when not top-ranked by conventional scores, and that VS is sensitive to over- vs under-prediction and forecast timing. The work highlights the importance of context-aware evaluation for decision-makers and discusses extensions to more realistic losses and larger datasets. Overall, the VS framework offers a practical, decision-centric metric to guide the utilization and comparison of epidemic forecasts in policy settings.

Abstract

In order for epidemiological forecasts to be useful for decision-makers the forecasts need to be properly validated and evaluated. Although several metrics fore evaluation have been proposed and used none of them account for the potential costs and losses that the decision-maker faces. We have adapted a decision-theoretic framework to an epidemiological context which assigns a Value Score (VS) to each model by comparing the expected expense of the decision-maker when acting on the model forecast to the expected expense obtained from acting on historical event probabilities. The VS depends on the cost-loss ratio and a positive VS implies added value for the decision-maker whereas a negative VS means that historical event probabilities outperform the model forecasts. We apply this framework to a subset of model forecasts of influenza peak intensity from the FluSight Challenge and show that most models exhibit a positive VS for some range of cost-loss ratios. However, there is no clear relationship between the VS and the original ranking of the model forecasts obtained using a modified log score. This is in part explained by the fact that the VS is sensitive to over- vs. under-prediction, which is not the case for standard evaluation metrics. We believe that this type of context-sensitive evaluation will lead to improved utilisation of epidemiological forecasts by decision-makers.

Figures (6)

• Figure 1: An example of a probabilistic forecast of the wILI peak intensity made at epiweek 47 when wILI first crosses the seasonal baseline. The dashed line corresponds to the threshold value for a severe season and the orange bars show the probability of a severe season, which in this case equals $p_f=0.03$.
• Figure 2: A schematic of the cost-loss framework as applied to the FluSight Challenge data. Each panel shows the weekly wILI over an influenza season, and the inset shows the forecasted probability distribution of the peak intensity. At the first week when wILI exceeds the baseline (dashed line) the forecaster issues a probabilistic prediction of the wILI peak intensity. If the forecasted probability that peak intensity exceeds the severity thresholds $TI_{90}$ (red solid line) is larger than the cost-loss threshold, which in this example is assumed to be $C/L=0.4$, the decision-maker prepares for a severe season. In this case the expense is given by $C$ independent if the peak intensity exceeds the threshold or not (panel A and B). If the forecasted probability is less than $C/L$ no action is taken, and in the absence of a severe season the expense is zero (panel D). However, if the threshold is exceeded a loss $L$ is incurred (panel C).
• Figure 3: Value Score of forecasts from the FluSight Challenge including the Target-Type Weights (TTW) ensemble and the ReichLab-KDE, which is a historical baseline model. The Value Score is calculated relative to a historical baseline event probability of $p_b=0.32$. The VS is plotted against the cost-loss ratio $C/L$. A $\hbox{VS}>0$ for a given $C/L$ implies that a decision-maker who is trying to minimise their expected expense is to prefer the model forecast over the baseline, whereas $\hbox{VS}<0$ suggests that acting on the baseline model is preferable.
• Figure 4: The Value Score of the A) CU-EAKFC-SIRS-model and B) the LANL-DBM-model for forecasts of wILI peak intensity exceeding $TI_{90}$ made at season onset and 1-5 weeks later.
• Figure 5: The Value Score of model A relative to B for CU-EAKFC-SIRS, CU-BMA, Delphi-Density1 and TTW-ensemble. Each line (red dashed or solid blue) corresponds to that model relative to the other model in that panel. Note that the value score of model A relative to B is undefined when model B produces forecasts on par with the perfect forecast.