Behavior of prediction performance metrics with rare events

TL;DR

This study investigates how rare-event contexts affect prediction performance metrics, with a focus on AUC. Using plasmode simulations based on a large mental health dataset (event rate $R_0 = 0.92\%$) and three modeling approaches (GLM, ridge, RF), the authors show that bias and variance in AUC, as well as in sensitivity and specificity, are driven by the effective sample size (number of events or non-events) rather than the event rate itself; approximately 1000 events suffice for near-zero AUC bias. Other metrics such as PPV and accuracy remain influenced by the event rate, though their bias also improves with larger ESS. The work argues for reporting multiple metrics with uncertainty estimates in rare-event settings and highlights logistic regression stability issues when events per variable are low, reinforcing the importance of adequate ESS and model selection in high-stakes, rare-outcome predictions.

Abstract

Objective: Area under the receiving operator characteristic curve (AUC) is commonly reported alongside prediction models for binary outcomes. Recent articles have raised concerns that AUC might be a misleading measure of prediction performance in the rare event setting. This setting is common since many events of clinical importance are rare. We aimed to determine whether the bias and variance of AUC are driven by the number of events or the event rate. We also investigated the behavior of other commonly used measures of prediction performance, including positive predictive value, accuracy, sensitivity, and specificity. Study Design and Setting: We conducted a simulation study to determine when or whether AUC is unstable in the rare event setting by varying the size of datasets used to train and evaluate prediction models. This plasmode simulation study was based on data from the Mental Health Research Network; the data contained 149 predictors and the outcome of interest, suicide attempt, which had event rate 0.92\% in the original dataset. Results: Our results indicate that poor AUC behavior -- as measured by empirical bias, variability of cross-validated AUC estimates, and empirical coverage of confidence intervals -- is driven by the number of events in a rare-event setting, not event rate. Performance of sensitivity is driven by the number of events, while that of specificity is driven by the number of non-events. Other measures, including positive predictive value and accuracy, depend on the event rate even in large samples. Conclusion: AUC is reliable in the rare event setting provided that the total number of events is moderately large; in our simulations, we observed near zero bias with 1000 events.

Figures (6)

• Figure 1: Empirical bias and coverage of 95% confidence intervals for estimating the evaluation-set AUC (values provided in Table \ref{['tab:true_aucs']}) versus effective sample size (number of events in the training set) in the rows; columns show logistic regression (GLM) including all predictors, random forests (RF), and ridge logistic regression (Ridge) including all predictors. Estimates from the training dataset were compared to true values computed on the evaluation dataset. Colors and shapes show the training set size; for each training set size, an increasing event rate leads to a larger effective sample size. ESE = empirical standard error, ASE = asymptotic standard error. The blue dashed lines around 95% coverage indicate one Monte-Carlo standard error.
• Figure 2: Empirical bias and coverage of 95% confidence intervals for estimating the evaluation-set sensitivity at the $95^\text{th}$ percentile of predicted risk versus effective sample size, using logistic regression (GLM), random forests (RF), and ridge regression (Ridge). Effective sample size for sensitivity is the number of events in the training set; for specificity, it is the number of non-events in the training set. Estimates from the training dataset were compared to true values computed on the evaluation dataset. Colors and shapes show the training set size; at each fixed number of events, the larger training set size implies a smaller event rate. The blue dashed lines around 95% coverage indicate one Monte-Carlo standard error.
• Figure 3: Empirical bias and coverage of 95% confidence intervals for estimating the evaluation-set specificity at the 95$^\text{th}$ percentile of predicted risk versus effective sample size, using logistic regression (GLM), random forests (RF), and ridge regression (Ridge). Effective sample size for sensitivity is the number of events in the training set; for specificity, it is the number of non-events in the training set. Estimates from the training dataset were compared to true values computed on the evaluation dataset. Colors and shapes show the training set size; at each fixed number of events, the larger training set size implies a smaller event rate. The blue dashed lines around 95% coverage indicate one Monte-Carlo standard error.
• Figure 4: Empirical bias and coverage of 95% confidence intervals for estimating the evaluation-set PPV at the 95th percentile of predicted risk versus effective sample size (number of events in the training set) using logistic regression (GLM), random forests (RF), and ridge regression (Ridge). Estimates from the training dataset were compared to true values computed on the evaluation dataset. Colors and shapes show the training set size; at each fixed number of events, the larger training set size implies a smaller event rate. The blue dashed lines around 95% coverage indicate one Monte-Carlo standard error.
• Figure S1: Empirical bias and coverage of 95% confidence intervals for estimating the evaluation-set AUC (values provided in Table 3) versus training set size at the three event rates (rows); columns show logistic regression (GLM) including all predictors, random forests (RF), and ridge logistic regression (Ridge) including all predictors. ESE = empirical standard error, ASE = asymptotic standard error. The blue dashed lines around 95% coverage indicate one Monte-Carlo standard error.