Comparing regularisation paths of (conjugate) gradient estimators in ridge regression

TL;DR

The paper develops a non-asymptotic framework to compare regularisation paths of gradient-based methods for ridge regression under random design. By decomposing errors via residual polynomials and introducing a data-dependent time reparametrisation $\tau_t$, it shows CG iterates along their path have prediction risk bounded by the GF path, and hence by ridge regression, up to constants depending on the spectrum of $\widehat{\Sigma}$; an oracle-type bound connects CG to GF and RR. Numerical experiments demonstrate that CG, GF, and RR share almost indistinguishable regularisation paths, with CG achieving minimal risk much faster, indicating strong practical and theoretical guarantees for CG with early stopping in high-dimensional settings. The results rely on a novel CG-specific error decomposition and extend GF/RR comparisons to the data-dependent Krylov subspaces generated by CG, offering guidance for stopping rules and hyperparameter tuning in practice.

Abstract

We consider standard gradient descent, gradient flow and conjugate gradients as iterative algorithms for minimising a penalised ridge criterion in linear regression. While it is well known that conjugate gradients exhibit fast numerical convergence, the statistical properties of their iterates are more difficult to assess due to inherent non-linearities and dependencies. On the other hand, standard gradient flow is a linear method with well-known regularising properties when stopped early. By an explicit non-standard error decomposition we are able to bound the prediction error for conjugate gradient iterates by a corresponding prediction error of gradient flow at transformed iteration indices. This way, the risk along the entire regularisation path of conjugate gradient iterations can be compared to that for regularisation paths of standard linear methods like gradient flow and ridge regression. In particular, the oracle conjugate gradient iterate shares the optimality properties of the gradient flow and ridge regression oracles up to a constant factor. Numerical examples show the similarity of the regularisation paths in practice.

Figures (2)

• Figure 1: Left: Filters $R_{t}^{\mathrm{CG}}$ (orange), $R_{\rho_t}^{\mathrm{GF}}$ (blue), $R_{\lambda,\lambda+1/\rho_t}^{\mathrm{RR}}$ (red) and the first zero $x_{1,t}$ of $R_{t}^{\mathrm{CG}}$ (dashed vertical). Right: Filters $(1-R_{t,<}^{\mathrm{CG}})^{1/2}$ (orange), $1-R_{\rho_t}^{\mathrm{GF}}$ (blue), $1-R_{\lambda,\lambda+1/\rho_t}^{\mathrm{RR}}$ (red) for the stochastic errors.
• Figure 2: Left: Prediction risk in simulation example for $\gamma=\beta_0$ (solid lines) and $\gamma=\beta_\lambda$ (dashed lines). The grey lines represent the terminal risk of the corresponding ridge estimator $\hat{\beta}^{\mathrm{RR}}_{\lambda}$. Right: Out-of-sample ridge criterion for the Riboflavin data set and the three estimation methods.

Theorems & Definitions (20)

• Proposition 3.1: Standard error decomposition
• proof
• proof
• Proposition 3.3
• Corollary 3.4
• Proposition 3.5
• proof
• Corollary 3.6
• proof
• Theorem 3.7