ROTI-GCV: Generalized Cross-Validation for right-ROTationally Invariant Data

TL;DR

This work introduces ROTI-GCV, a cross-validation framework tailored for ridge regression with right-rotationally invariant designs that capture sample dependence and heavy tails. By showing that standard GCV is biased under these designs, the authors derive a consistent estimating-equation approach to infer the signal-to-noise ratio $r^2$ and noise variance $\sigma^2$, then form a plug-in cross-validation metric that tracks the true out-of-sample risk uniformly on compact $\lambda$-ranges. The paper further handles signal-PC alignment with an aROTI-GCV variant that accounts for aligned and coupled eigenvectors, providing consistent risk estimation in more structured settings. Extensive simulations and semi-real data (e.g., speech and financial residuals) demonstrate the practical effectiveness and robustness of ROTI-GCV and aROTI-GCV in challenging covariate regimes, highlighting their potential for reliable hyperparameter tuning beyond the i.i.d. paradigm.

Abstract

Two key tasks in high-dimensional regularized regression are tuning the regularization strength for accurate predictions and estimating the out-of-sample risk. It is known that the standard approach -- $k$-fold cross-validation -- is inconsistent in modern high-dimensional settings. While leave-one-out and generalized cross-validation remain consistent in some high-dimensional cases, they become inconsistent when samples are dependent or contain heavy-tailed covariates. As a first step towards modeling structured sample dependence and heavy tails, we use right-rotationally invariant covariate distributions -- a crucial concept from compressed sensing. In the proportional asymptotics regime where the number of features and samples grow comparably, which is known to better reflect the empirical behavior in moderately sized datasets, we introduce a new framework, ROTI-GCV, for reliably performing cross-validation under these challenging conditions. Along the way, we propose new estimators for the signal-to-noise ratio and noise variance. We conduct experiments that demonstrate the accuracy of our approach in a variety of synthetic and semi-synthetic settings.

Figures (8)

• Figure 1: Risk curves produced by the cross-validation methods in addition to the true risk curve. Each plot uses a different form of right-rotationally invariant data, as indicated in each caption. (\ref{['fig:auto-gcv']}): Autocorrelated data: rows of $\mathbf{X}$ are drawn according to $\mathbf{x}_i = \rho \mathbf{x}_{i - 1} + \sqrt{1 - \rho^2} \mathbf{z}_i$, with $\mathbf{z}_i$ being i.i.d. draws from $\mathsf{N}(0, \mathbf{I}_n)$. We set $\rho = 0.8$. (\ref{['fig:t-gcv']}): $t$-distributed data: each row is drawn from a multivariate $t$ distribution with $3$ degrees of freedom. (\ref{['fig:equi-gcv']}): Equicorrelated data $\mathbf{X} \in \mathbb{R}^{n \times p}$ has independent columns, but each column follows a multivariate Gaussian distribution with covariance matrix $\boldsymbol\Sigma$, where $\Sigma_{ij} = \rho$ if $i \neq j$, and $\Sigma_{ii} = 1$. All simulations have $n = p = 1000$ and $r^2 = \sigma^2 = 1$. The $x$-axis for every plot is $\lambda$, the regularization parameter. The colored lines (blue, green, red, purple) are one of 10 iterations. In each iteration, we compute the cross-validation metric over a range of $\lambda$ to produce the line, which reflects the estimated out-of-sample risk. The black line of each plot shows the average result for that method. The dashed blue line shows the average expected MSE curve as a benchmark.
• Figure 2: As in Figure \ref{['fig:gcv-synth-perf']}, curves show the risk and estimated risk as a function of $\lambda$. (\ref{['fig:aligned-gcv']}): $\mathbf{X}$ has autocorrelated rows, with $\rho = 0.8$. The top 10 eigenvectors ($\mathcal{J}_c = [10]$) are coupled, so that the top 10 eigenvectors of the test set are equal to those of $\mathbf{X}$. $\boldsymbol\beta = \boldsymbol\beta' + \sqrt{n}\sum_{i = 1} ^{10} \frac{i}{10} \mathbf{o}_i$, with $\|\boldsymbol\beta'\|^2 = \sqrt{n}$, $\sigma^2 = 1$, $n = p = 1000$. (\ref{['fig:mixture']}) i.i.d. rows drawn from a Gaussian mixture, i.e. $\mathbf{x}_i$ is drawn from $\frac{1}{2} \mathsf{N}(\mathbf{3}, \mathbf{I}_p) + \frac{1}{2} \mathsf{N}(-\mathbf{3}, \mathbf{I}_p)$. When computing ROTI-GCV, we set $\mathcal{J}_c = \{1\}$. (\ref{['fig:row-equi']}): i.i.d. rows drawn from an equicorrelated Gaussian, i.e. $\mathbf{x}_i \sim \mathsf{N}(0, \boldsymbol\Sigma)$, where $\Sigma_{ij} = \mathds{1}(i = j) + \rho\mathds{1}(i \neq j)$. We set $\rho = 0.5$. When computing ROTI-GCV, we set $\mathcal{J}_c = [10]$. (\ref{['fig:speech-data-result']}): speech data with $\boldsymbol\beta$ sampled uniformly from sphere, with $r^2 = \sigma^2 = 1$, $n = p = 400$; we choose $\mathcal{J}_c = [3]$. (\ref{['fig:speech-data-align']}): speech data with $\boldsymbol\beta = \boldsymbol\beta' + \tfrac{\sqrt{n}}{2}\sum_{i = 1} ^5 \mathbf{o}_i$; again $r^2 = \sigma^2 = 1$ and $n = p = 400$. We choose $\mathcal{J}_c = [3]$ and $\mathcal{J}_a = [5]$ (discussion in Section \ref{['sec:ex-verify']}). (\ref{['fig:sp500']}): 30 minute residualized returns sampled every 1 minute; $\boldsymbol\beta$ sampled uniformly from sphere.
• Figure 3: Singular value distribution for speech data.
• Figure 4: Residualized returns setting, $n = p = 493$, $r^2 = \sigma^2 = 1$; Figure \ref{['fig:resid-ret-overlaps']} is essentially a histogram of values in \ref{['fig:resid-ret-numericals']}.
• Figure 5: Gaussian mixture setting, $n = p = 1000$; Figure \ref{['fig:gaussian-mixture-overlaps']} is essentially a histogram of values in \ref{['fig:gaussian-mixture-numericals']}.

Theorems & Definitions (29)

• Definition 1: Right-rotationally invariant design
• Remark 1
• Remark 2
• Theorem 1
• Theorem 2
• Remark 3
• Lemma 1
• Corollary 2
• Corollary 3
• Remark 4