Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated Kaczmarz methods via randomized sketch techniques for solving consistent linear systems

Haochen Jiang, Dongdong Liu, Xianping Wu, Xu Yang

TL;DR

This paper addresses solving consistent linear systems by accelerating Kaczmarz-type methods through randomized sketches. It introduces two sketch-based variants, RS-MWRK(S) and LS-RaBK(S), leveraging two random matrices to reduce computational cost per iteration while maintaining convergence. The authors provide convergence guarantees via subspace embedding properties, quantify contraction rates in terms of $\sigma_r(A)$ and other spectral quantities, and bound deviations between the accelerated and original methods. Empirical results on extremely large-scale problems show substantial runtime reductions, demonstrating practical impact for high-dimensional least-squares problems.

Abstract

Motivated by the randomized sketch to solve a variety of problems in scientific computation, we improve both the maximal weighted residual Kaczmarz method and the randomized block average Kaczmarz method using two new randomized sketch techniques. Besides, convergence analyses of the proposed methods are provided. Furthermore, we establish an upper bound for the discrepancy between the numerical solutions obtained via the proposed methods and those derived from the original approaches. Numerical experiments demonstrate that the new methods perform better than the existing ones in terms of the running time with the same accuracy.

Accelerated Kaczmarz methods via randomized sketch techniques for solving consistent linear systems

TL;DR

This paper addresses solving consistent linear systems by accelerating Kaczmarz-type methods through randomized sketches. It introduces two sketch-based variants, RS-MWRK(S) and LS-RaBK(S), leveraging two random matrices to reduce computational cost per iteration while maintaining convergence. The authors provide convergence guarantees via subspace embedding properties, quantify contraction rates in terms of and other spectral quantities, and bound deviations between the accelerated and original methods. Empirical results on extremely large-scale problems show substantial runtime reductions, demonstrating practical impact for high-dimensional least-squares problems.

Abstract

Motivated by the randomized sketch to solve a variety of problems in scientific computation, we improve both the maximal weighted residual Kaczmarz method and the randomized block average Kaczmarz method using two new randomized sketch techniques. Besides, convergence analyses of the proposed methods are provided. Furthermore, we establish an upper bound for the discrepancy between the numerical solutions obtained via the proposed methods and those derived from the original approaches. Numerical experiments demonstrate that the new methods perform better than the existing ones in terms of the running time with the same accuracy.

Paper Structure

This paper contains 8 sections, 6 theorems, 27 equations, 1 figure, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The proposed methods
  4. Accelerated Kaczmarz methods with randomized matrix
  5. Selection of the randomized matrix $\mathbf{S}$
  6. Convergence analysis
  7. Numerical experiments
  8. Conclusions

Key Result

Lemma 2.1

(meng, Theorem 1) For any matrix $A\in \mathbb{R}^{m\times n}$, $\mathbf{S}$ is an $d\times m$ matrix that embeds the column space of $A$ into $\mathbb{R}^d$ while approximately preserving the $\ell_2$ norms of all vectors in that subspace. Then we have that

Figures (1)

  • Figure 5.1: $log_{10}(RES)$ versus IT and CPU for CS-MWRK, RS-MWRK(Q), RS-MWRK(G), LS-RaBK(Q)-c, RaBK-c, LS-RaBK(Q)-a and RaBK-a when $A\in\mathbb{R}^{5000000\times 50}$.

Theorems & Definitions (9)

  • Remark 2.1
  • Lemma 2.1
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4
  • Lemma 2.5
  • Remark 3.1
  • Remark 3.2
  • Lemma 4.1