Fast and forward stable randomized algorithms for linear least-squares problems

TL;DR

This work resolves whether randomized least-squares solvers can be both faster than Householder QR and numerically stable by proving that carefully implemented iterative sketching is forward stable. It introduces a numerically stable implementation that uses a single subspace embedding and an economy QR for robust triangular solves, and analyzes the propagation of errors through iterative refinement on the normal equations. Theoretical results (forward-stability bounds) are complemented by extensive numerical experiments on dense kernel regression and large sparse LS problems, showing substantial speedups over QR while achieving residual accuracy on par with backward-stable methods within the forward-stability regime. The findings demonstrate that randomized iterative sketching can outperform traditional QR-based approaches in large-scale, overdetermined LS problems, and they offer practical guidance for deployment, including embedding choices, stopping criteria, and damping/momentum variants.

Abstract

Iterative sketching and sketch-and-precondition are randomized algorithms used for solving overdetermined linear least-squares problems. When implemented in exact arithmetic, these algorithms produce high-accuracy solutions to least-squares problems faster than standard direct methods based on QR factorization. Recently, Meier, Nakatsukasa, Townsend, and Webb demonstrated numerical instabilities in a version of sketch-and-precondition in floating point arithmetic (arXiv:2302.07202). The work of Meier et al. raises the question: Is there a randomized least-squares solver that is both fast and stable? This paper resolves this question in the affirmative by proving that iterative sketching, appropriately implemented, is forward stable. Numerical experiments confirm the theoretical findings, demonstrating that iterative sketching is stable and faster than QR-based solvers for large problem instances.

Figures (8)

• Figure 1: Forward (left) and backward (right) error for iterative sketching (solid lines). We consider three different condition numbers $\kappa = 10^1$ (blue triangles), $\kappa = 10^{10}$ (red squares), and $\kappa = 10^{15}$ (yellow circles) and two residuals $\mleft\| \boldsymbol{r}(\boldsymbol{x}) \mright\| = 10^{-12}$ (top) and $\mleft\| \boldsymbol{r}(\boldsymbol{x}) \mright\| = 10^{-3}$ (bottom). Reference accuracies for Householder QR are shown as dotted lines.
• Figure 2: Forward (left) and backward (right) error for sketch-and-precondition (S&P) and iterative sketching (IS, yellow dotted) with $\kappa = 10^{10}$ and $\mleft\| \boldsymbol{r}(\boldsymbol{x}) \mright\| = 10^{-6}$. We consider sketch-and-precondition with both the zero initialization (red dashed) and the sketch-and-solve initialization (blue solid). Reference errors for Householder QR are shown as black dotted lines.
• Figure 3: Comparison of the stable implementation of iterative sketching (\ref{['alg:it-sk']}, blue solid) to three "bad" implementations, referred to as bad matrix (red dash-dotted), bad residual (yellow dotted), and bad initial (purple dashed). The accuracy of Householder QR factorization is provided as a baseline (black dotted).
• Figure 4: Left: Runtime of iterative sketching and MATLAB's mldivide for the kernel regression problem for numbers of columns $n \in [10^1,10^3]$. Right: Relative difference of solution $\boldsymbol{\widehat{x}}$ by iterative sketching and $\boldsymbol{\widehat{x}}_{\rm dir}$ by MATLAB's mldivide. The upper bound provided by Wedin's theorem (\ref{['thm:wedin']}) with $\varepsilon = u$ is shown as a dotted maroon line.
• Figure 5: Runtime (left) and memory usage (right) of iterative sketching and MATLAB's mldivide for the sparse problem for numbers of columns $n \in [10^1,10^3]$.

Theorems & Definitions (10)

• Definition 2: Subspace embedding
• Theorem 5: Convergence of iterative sketching
• Theorem 6: Iterative sketching is forward stable
• Proposition 7: Singular value bounds, floating point
• proof
• Lemma 8: Bound on initial error
• proof
• Proposition 9: Solution to recurrence
• Theorem 10: Damping and momentum
• proof