Preconditioning without a preconditioner: faster ridge-regression and Gaussian sampling with randomized block Krylov subspace methods
Tyler Chen, Caroline Huber, Ethan Lin, Hajar Zaid
TL;DR
This work introduces a randomized variant of the block conjugate gradient method that achieves faster convergence for solving regularized linear systems $A_bmu x=b$ without explicitly constructing a preconditioner. By augmenting the starting block with random sketches $\mathbf{\Omega}$, the method induces implicit preconditioning within the block-Krylov subspace, enabling guarantees that compare favorably to Nyström-based preconditioners while reducing matrix-load costs. The authors derive explicit probabilistic bounds and improved matrix-vector product complexities, enabling efficient computation of the entire ridge-regression path and fast Gaussian sampling from $\mathcal{N}(\mu, A)$. They support their theory with extensive numerical experiments showing gains in convergence speed and sampling accuracy, along with discussions on practical considerations such as reorthogonalization and block-size effects. The approach provides a new lens on block-Krylov methods, suggesting broad applicability to regression and matrix-function tasks beyond standard preconditioning frameworks.
Abstract
We describe a randomized variant of the block conjugate gradient method for solving a single positive-definite linear system of equations. Our method provably outperforms preconditioned conjugate gradient with a broad-class of Nyström-based preconditioners, without ever explicitly constructing a preconditioner. In analyzing our algorithm, we derive theoretical guarantees for new variants of Nyström preconditioned conjugate gradient which may be of separate interest. We also describe how our approach yields state-of-the-art algorithms for key data-science tasks such as computing the entire ridge regression regularization path and generating multiple independent samples from a high-dimensional Gaussian distribution.