Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomized Iterative Solver as Iterative Refinement: A Simple Fix Towards Backward Stability

Ruihan Xu, Yiping Lu

TL;DR

A new perspective is introduced that interprets Iterative Sketching and Sketching-and-Precondition as forms of Iterative Refinement, and proposes a novel algorithm, Sketched Iterative and Recursive Refinement (SIRR), which combines both refinement methods.

Abstract

Iterative sketching and sketch-and-precondition are well-established randomized algorithms for solving large-scale, over-determined linear least-squares problems. In this paper, we introduce a new perspective that interprets Iterative Sketching and Sketching-and-Precondition as forms of Iterative Refinement. We also examine the numerical stability of two distinct refinement strategies, iterative refinement and recursive refinement, which progressively improve the accuracy of a sketched linear solver. Building on this insight, we propose a novel algorithm, Sketched Iterative and Recursive Refinement (SIRR), which combines both refinement methods. SIRR demonstrates a \emph{four order of magnitude improvement} in backward error compared to iterative sketching, achieved simply by reorganizing the computational order, ensuring that the computed solution exactly solves a modified least-squares system where the coefficient matrix deviates only slightly from the original matrix. To the best of our knowledge, \emph{SIRR is the first asymptotically fast, single-stage randomized least-squares solver that achieves both forward and backward stability}.

Randomized Iterative Solver as Iterative Refinement: A Simple Fix Towards Backward Stability

TL;DR

A new perspective is introduced that interprets Iterative Sketching and Sketching-and-Precondition as forms of Iterative Refinement, and proposes a novel algorithm, Sketched Iterative and Recursive Refinement (SIRR), which combines both refinement methods.

Abstract

Iterative sketching and sketch-and-precondition are well-established randomized algorithms for solving large-scale, over-determined linear least-squares problems. In this paper, we introduce a new perspective that interprets Iterative Sketching and Sketching-and-Precondition as forms of Iterative Refinement. We also examine the numerical stability of two distinct refinement strategies, iterative refinement and recursive refinement, which progressively improve the accuracy of a sketched linear solver. Building on this insight, we propose a novel algorithm, Sketched Iterative and Recursive Refinement (SIRR), which combines both refinement methods. SIRR demonstrates a \emph{four order of magnitude improvement} in backward error compared to iterative sketching, achieved simply by reorganizing the computational order, ensuring that the computed solution exactly solves a modified least-squares system where the coefficient matrix deviates only slightly from the original matrix. To the best of our knowledge, \emph{SIRR is the first asymptotically fast, single-stage randomized least-squares solver that achieves both forward and backward stability}.

Paper Structure

This paper contains 49 sections, 8 theorems, 88 equations, 6 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Contribution
  4. Preliminary
  5. Sketch-and-Precondition
  6. Iterative Hessian Sketching pilanci2016iterativeozaslan2019iterative
  7. Backward-Stability
  8. Numerical Stability
  9. Randomized Solver As Iterative Refinement
  10. Iterative and Recursive refinement
  11. Iterative Refinement
  12. Recursive Refinement
  13. Recursive Refinement as Reorganizing Computation
  14. Equivalence between Iterative Refinement and Sketch-and-Precondition
  15. Convergence of Iterative and Recursive refinement
  16. ...and 34 more sections

Key Result

Lemma 1

For matrix $A\in \mathbb{R}^{m\times n}$, there exists sketching matrix $S\in \mathbb{R}^{s\times m}$. Suppose that $\hat{R}\hat{Q}=SA$ is the QR decomposition of matrix $SA$, then the following inequalities holds:

Figures (6)

  • Figure 1: Results of SIRR with sketch and solve Initialization are shown as solid curve lines, with reference accuracy for MATLAB function A$\backslash$b shown as dotted constant lines and IHS-Krylov shown as dotted curve lines
  • Figure 2: Forward error (left) and backward error (right) under different $\|b-Ax^\ast\|/\|Ax^\ast\|$. SRR is not backward stable when $\|b-Ax^\ast\|$ is small while SIRR can achieve backward stable estimates for all cases. We also plotted the result for mldivide(MATLAB) solver here for reference.
  • Figure 3: Comparing the Forward error (left) and backward error (right) of SIRR and FOSSILS on problems with different difficulties. SIRR has better backward stability in most situations and similar forward stability compared to FOSSILS.
  • Figure 4: Forward error (left) and backward error (right) of different sizes of $n$.
  • Figure 5: In first row $\|b-Ax^\ast\|=10^{-1}$ and in second row $\|b-Ax^\ast\|=10^{-3}$ with $\kappa = 10^4,10^8,10^{12}$ from left to right.
  • ...and 1 more figures

Theorems & Definitions (17)

  • Lemma 1: higham2002accuracymeier2024sketchepperly2023fastepperly2024fast
  • Definition : Backward error
  • Definition : $\alpha-\beta$ Accuracy
  • Lemma 2
  • Theorem 3: Convergence of Iterative/Recursive Refinement
  • Remark 1: Selection of Meta-Algorithm
  • Theorem 4: Convergence of Iterative/Recursive Refinement
  • Remark 2: Selection of Meta-Algorithm
  • Definition : Forward Stability
  • Remark 3
  • ...and 7 more