Stability Regularized Cross-Validation

TL;DR

The paper tackles the adaptivity gap between cross-validation and test performance by introducing stability-regularized nested cross-validation, which optimizes a weighted sum of CV error and an empirical stability measure. A generalization bound is extended from leave-one-out to k-fold CV, motivating a stability term in model selection; a nested CV scheme selects the trade-off weight to reduce out-of-sample disappointment. Empirical evaluation on 13 real-world datasets shows 4% average improvements for sparse ridge regression and CART, while XGBoost remains largely unaffected, highlighting practical gains for interpretable but unstable models. Overall, the approach provides a computationally feasible method to improve test-set performance and reliability of CV-based hyperparameter selection, with potential for broader applicability and tighter theoretical guarantees in future work.

Abstract

We revisit the problem of ensuring strong test-set performance via cross-validation. Motivated by the generalization theory literature, we propose a nested k-fold cross-validation scheme that selects hyperparameters by minimizing a weighted sum of the usual cross-validation metric and an empirical model-stability measure. The weight on the stability term is itself chosen via a nested cross-validation procedure. This reduces the risk of strong validation set performance and poor test set performance due to instability. We benchmark our procedure on a suite of 13 real-world UCI datasets, and find that, compared to k-fold cross-validation over the same hyperparameters, it improves the out-of-sample MSE for sparse ridge regression and CART by 4% on average, but has no impact on XGBoost. This suggests that for interpretable and unstable models, such as sparse regression and CART, our approach is a viable and computationally affordable method for improving test-set performance.

Figures (2)

• Figure 1: Leave-one-out (LOOCV, left) and test (right) error for varying $\tau$ and $\gamma$, for an overdetermined setting (top, $n=50, p=10$) and an underdetermined setting (bottom, $n=10, p=50$). In the overdetermined setting, LOOCV is a good estimate of the test error for most values of parameters $(\gamma,\tau)$. In contrast, in the underdetermined setting, LOOCV is a poor approximation of the test error, and estimators that minimize LOOCV ($\gamma\to 0$, $\tau=10$) significantly disappoint out-of-sample. Our conclusions are identical when using five-fold cross-validation (Appendix \ref{['sec.append:heatmap']}).
• Figure 2: Five-fold (left) and test (right) error for varying $\tau$ and $\gamma$, for the overdetermined setting (top, $n=50, p=10$) and an underdetermined setting (bottom, $n=10, p=50$) considered in Figure \ref{['fig:l10cv_test_comp']}. In the overdetermined setting, the five-fold error is a good estimate of the test error for most values of parameters $(\gamma,\tau)$. In contrast, in the underdetermined setting, the five-fold error is a poor approximation of the test error, and the estimator that minimizes the five-fold error ($\gamma= 6.15$, $\tau=5$) significantly disappoint out-of-sample.

Theorems & Definitions (5)

• Proposition 1
• proof
• Theorem 1
• Remark 1: To Train or to Validate in \ref{['l10ucb']}
• proof