Uniform Consistency of Generalized Cross-Validation for Ridge Regression in High-Dimensional Misspecified Linear Models
Akira Shinkyu
TL;DR
The paper analyzes tuning parameter selection for ridge regression in high-dimensional settings when the true regression function is misspecified as linear. It extends generalized cross-validation (GCV) to include zero and negative tuning values and proves uniform consistency for the out-of-sample risk under the condition that the second moment of the specification error vanishes, yielding asymptotically optimal tuning within the candidate set. The main result shows that GCV (and LOOCV under mean-zero responses) can asymptotically achieve minimal prediction risk in misspecified settings. Simulations demonstrate that ridge regression tuned by GCV closely matches optimally tuned ridge and outperforms Lasso under both correct and incorrect specifications, highlighting GCV’s practical robustness in high dimensions.
Abstract
This study examines generalized cross-validation for the tuning parameter selection for ridge regression in high-dimensional misspecified linear models. The set of candidates for the tuning parameter includes not only positive values but also zero and negative values. We demonstrate that if the second moment of the specification error converges to zero, generalized cross-validation is still a uniformly consistent estimator of the out-of-sample prediction risk. This implies that generalized cross-validation selects the tuning parameter for which ridge regression asymptotically achieves the smallest prediction risk among the candidates if the degree of misspecification for the regression function is small. Our simulation studies show that ridge regression tuned by generalized cross-validation exhibits a prediction performance similar to that of optimally tuned ridge regression and outperforms the Lasso under correct and incorrect model specifications.