On the calibration of survival models with competing risks

TL;DR

This work tackles the gap in calibrated probability estimates for survival models with competing risks by introducing two proper calibration notions: CR D-calibration, a distribution-based metric extending probability integral transforms to competing events with censoring, and cal_{K}^{\alpha}-calibration, a marginal CIF-based calibration measure. The authors provide consistent estimators, statistical tests, and two post-hoc recalibration methods (AJ-recalibration and competing-risks temperature scaling) that improve calibration without reducing discrimination (C-index). They establish theoretical properties, including the properness of CR D-calibration and asymptotic consistency for marginal estimators, and demonstrate empirical gains on synthetic, SEER, and METABRIC datasets. The practical impact lies in delivering reliable, time-dependent, multi-class probability estimates critical for decision-making in healthcare and other domains involving competing risks. Overall, the framework provides rigorous calibration tools and actionable recalibration procedures that enhance the faithfulness of individualized CIFs while preserving model discrimination.

Abstract

Survival analysis deals with modeling the time until an event occurs, and accurate probability estimates are crucial for decision-making, particularly in the competing-risks setting where multiple events are possible. While recent work has addressed calibration in standard survival analysis, the competing-risks setting remains under-explored as it is harder (the calibration applies to both probabilities across classes and time horizon). We show that existing calibration measures are not suited to the competing-risk setting and that recent models do not give well-behaved probabilities. To address this, we introduce a dedicated framework with two novel calibration measures that are minimized for oracle estimators (i.e., both measures are proper). We also introduce some methods to estimate, test, and correct the calibration. Our recalibration methods yield good probabilities while preserving discrimination.

Figures (25)

• Figure 1: Marginal probabilities of SOTA models do not behave as expected: We compare the marginal probabilities of the different SOTA models on the SEER Dataset with 10k training samples. All these marginals should be close to the aalen_empirical_1978 estimator, which is marginally consistent.
• Figure 2: Existing calibration measures cast to the competing-risks setting: The CIF (Cumulative Incidence Function) for an individual specifies, for each event type, the probability of experiencing that event before time $\tau$. In this space, different existing calibration measures correspond to projecting the vector-valued CIF onto a specific "direction": 1-calibration (binary calibration at a fixed time), KM-calibration (survival/Kaplan–Meier calibration), multiclass CR calibration plots, or calibration of global (population-level) CIFs. Each of these captures only one marginal aspect of the problem and fails to assess the joint calibration of the full CIF vector across event types.
• Figure 3: CR D-calibration without censoring: For each event, we compute the individualized CIFs. Then, for each observed individual $\mathbf{x}_i$ with event $k$, we compute $F_k(t_i|\mathbf{x}_i)$ and $F_k(\infty|\mathbf{x}_i)$. Following theorem \ref{['thm:bigth']}, we compare the $F_k(t_i|\mathbf{x}_i)/F_k(\infty|\mathbf{x}_i)$ cumulative empirical distribution to an uniform cumulative distribution function. Summing this metric over all event types the results gives a measure of the CR D-calibration.
• Figure 4: IBS and calibration on METABRIC. Boxplots over 5 seeds of the integrated Brier score (IBS), AJ-${\rm cal}_{K}^\alpha$-calibration (with parameter $\alpha = 2$), and CR-D-calibration ($\alpha = 2$) for each model and its AJ-recalibration and temperature-scaling (TS) recalibration. Results on the synthetic and SEER datasets are reported in Appendix \ref{['sec:moreresults']}.
• Figure 5: Survival-analysis recalibration of good competing risk probabilities breaks them. On this real-life dataset (SEER, 10k training samples), we apply the D-recalibration method in survival analysis qi_conformalized_2024 to aalen_empirical_1978 by treating the risks independently (cause-specific). It incorrectly changes the marginal probabilities, though these were correct, as we used a consistent marginal estimator. The root of the problem is that survival-risk recalibration is applied to events independently, while their probabilities are tied gorfine_frailty-based_2011.

Theorems & Definitions (42)

• definition 1: Quantities of interest
• definition 7: CR D-calibration
• theorem 8: Properness
• proof : Proof sketch
• corollary 9
• definition 10: CR $\hat{D}^{CR}_\alpha$-calibration
• proposition 11: Consistency of the estimator
• proof : Proof Sketch
• definition 12: ${\rm cal}_{K}^\alpha$-calibration(s)
• definition 13: Plug-in calibrations