Uncertainty quantification via cross-validation and its variants under algorithmic stability
Nicolai Amann, Hannes Leeb, Lukas Steinberger
TL;DR
The paper advances conditional uncertainty quantification for CV-based prediction intervals in high-dimensional settings by proving asymptotic conservativeness of CV under algorithmic stability and stochastic boundedness, and, in the continuous case, asymptotic validity. It shows asymptotic equivalence of the Jackknife and Jackknife+ under these conditions, and extends the equivalence to CV and CV+. A novel Lévy gauge is introduced as a central analytical device, enabling precise control of distributional differences and quantile-based guarantees. The authors also establish the necessity of the imposed conditions and discuss extensions to risk measures and full-conformal prediction. Practically, the work suggests that CV+ provides robust guarantees beyond CV, particularly for non-stable algorithms, while the Jackknife family remains reliable in stable, high-dimensional contexts.
Abstract
Recently, there has been substantial interest in statistical guarantees for cross-validation (CV) methods of uncertainty quantification in statistical learning (cf. Barber et al. 2021a, Liang and Barber 2024, Steinberger and Leeb 2023). These guarantees should hold under minimal assumptions on the data generating process and conditional on the training data, because numerous predictions are usually computed based on one and the same training sample. We push this objective to the limit: We prove asymptotic conditional conservativeness of CV, that is, the probability of the actual coverage probability, conditional on the training data, undershooting its nominal level vanishes asymptotically, under minimal assumptions. In particular, we impose a stability condition, require that the prediction error is stochastically bounded, and show that neither condition can be dropped in general. By way of an asymptotic equivalence result, we also show that the closely related CV+ method of Barber et al. (2021a) provides exactly the same conditional statistical guarantees as CV in large samples, thereby extending the range of applicability of CV+ to the high-dimensional regime. We conclude that, in view of its marginal coverage guarantee, CV+ does indeed improve over simple CV. For our proofs we introduce a new concept called Lévy gauge, which can be of independent interest.