Shrinkage to Infinity: Reducing Test Error by Inflating the Minimum Norm Interpolator in Linear Models
Jake Freeman
TL;DR
This work analyzes over-parameterized linear regression under strongly anisotropic covariances in the diverging $d/n$ regime. It shows that inflating the minimum-norm interpolator $\theta_{MN}$ by a factor $c>1$, i.e., using $c\theta_{MN}$, can reduce the generalization error $G(\cdot)$ when $\beta$ is aligned with top eigen-directions of $\Sigma$, a phenomenon termed the Inflation Property. The paper establishes both additive and multiplicative improvements, with the latter requiring stronger eigenstructure assumptions and yielding a near-optimal inflation constant $c_{opt}$ that can be approximated by $c_{opt}\approx \frac{\frac{n}{d}\beta^T\Sigma^2\beta}{\frac{n^2}{d^2}\beta^T\Sigma^3\beta}$ under suitable scaling. A data-splitting approach constructs a consistent estimator $\theta_{ds}$ whose inflation $c^*$ achieves $G(c^*\theta_{ds})=o(1)$ in probability, linking Johnson–Lindenstrauss-type projections to anti-regularization phenomena. These results rely on novel Gaussian-projection bounds for anisotropic covariances, and they illuminate how implicit regularization (and even anti-regularization) can improve prediction in high-dimensional, highly structured settings.
Abstract
Hastie et al. (2022) found that ridge regularization is essential in high dimensional linear regression $y=β^Tx + ε$ with isotropic co-variates $x\in \mathbb{R}^d$ and $n$ samples at fixed $d/n$. However, Hastie et al. (2022) also notes that when the co-variates are anisotropic and $β$ is aligned with the top eigenvalues of population covariance, the "situation is qualitatively different." In the present article, we make precise this observation for linear regression with highly anisotropic covariances and diverging $d/n$. We find that simply scaling up (or inflating) the minimum $\ell_2$ norm interpolator by a constant greater than one can improve the generalization error. This is in sharp contrast to traditional regularization/shrinkage prescriptions. Moreover, we use a data-splitting technique to produce consistent estimators that achieve generalization error comparable to that of the optimally inflated minimum-norm interpolator. Our proof relies on apparently novel matching upper and lower bounds for expectations of Gaussian random projections for a general class of anisotropic covariance matrices when $d/n\to \infty$.