Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Momentum via Minimal Dual Function for Accelerating Randomized Sparse Kaczmarz

Lu Zhang, Jinchuan Zeng, Hongxia Wang, Hui Zhang

Abstract

Recently, the randomized sparse Kaczmarz method has been accelerated by designing heavy ball momentum adaptively via a minimal-error principle. In this paper, we develop a new adaptive momentum method based on the minimal dual function principle to go beyond the exact measurement restriction of the minimal-error principle. Moreover, by integrating the new adaptive momentum method with the quantile-based sampling, we introduce a general algorithmic framework, called quantile-based randomized sparse Kaczmarz with minimal dual function momentum, which provides a unified approach to exact, noisy, or corrupted linear systems. In addition, we utilize the discrepancy principle and monotone error as stopping rules for the proposed algorithm. Theoretically, we establish linear convergence in expectation of Bregman distance up to a finite horizon related to the contaminated level. At last, we provide numerical illustrations on simulated and real-world data to demonstrate the effectiveness of our proposed method.

Adaptive Momentum via Minimal Dual Function for Accelerating Randomized Sparse Kaczmarz

Abstract

Recently, the randomized sparse Kaczmarz method has been accelerated by designing heavy ball momentum adaptively via a minimal-error principle. In this paper, we develop a new adaptive momentum method based on the minimal dual function principle to go beyond the exact measurement restriction of the minimal-error principle. Moreover, by integrating the new adaptive momentum method with the quantile-based sampling, we introduce a general algorithmic framework, called quantile-based randomized sparse Kaczmarz with minimal dual function momentum, which provides a unified approach to exact, noisy, or corrupted linear systems. In addition, we utilize the discrepancy principle and monotone error as stopping rules for the proposed algorithm. Theoretically, we establish linear convergence in expectation of Bregman distance up to a finite horizon related to the contaminated level. At last, we provide numerical illustrations on simulated and real-world data to demonstrate the effectiveness of our proposed method.

Paper Structure

This paper contains 33 sections, 9 theorems, 78 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. The related work
  3. Randomized Kaczmarz method and its variants
  4. Quantile-based RK method and its variants
  5. The stopping rules
  6. Organization
  7. Notation and Preliminaries
  8. Notation
  9. Convex analysis tools
  10. The perturbed dual problem
  11. The proposed method
  12. The technical details
  13. The quantile statistics to detect corrupted equations
  14. The new principle for adaptively updating parameters $\alpha_k,w_k$
  15. The proposed method
  16. ...and 18 more sections

Key Result

Lemma 2.1

Consider the problem (eq1.2) with $b$ replaced by $\tilde{b}$, where the contaminated right-hand vector $\tilde{b}=b+b^*$ satisfies $\|\tilde{b}-b\|\leq \delta$. If the contaminated level $\delta$ is unknown, the perturbed dual problem of (eq1.2) is given by

Figures (4)

  • Figure 1: Relative error after 2000 iterations of Quantile-RaSK-MM for a range of $\beta$ and $q$
  • Figure 2: The performance of the minimal error (ME) rule and the discrepancy principle (DP) on Quantile-RaSK-MM for solving noisy or corrupted systems
  • Figure 3: The performance of different methods on real matrices in corrupted case
  • Figure 4: CT reconstructions by Quantile-RaSK-MM with the ME stopping rule. Rows (left-to-right): ground truth for $N=20$, reconstructions from corrupted systems for $N=20$, reconstructions from noisy systems for $N=20$; ground truth for $N=30$, reconstructions from corrupted systems for $N=30$, reconstructions from noisy systems for $N=30$.

Theorems & Definitions (18)

  • Definition 2.1
  • Example 2.1
  • Lemma 2.1
  • proof
  • Lemma 2.2: Descent Lemma
  • Lemma 2.3: Lemma 3.1, tondji2024acceleration
  • Remark 3.1
  • Remark 3.2
  • Lemma 4.1: Lemma 3.1, schopfer2019linear
  • Theorem 4.1: Theorem 3.2, schopfer2019linear; Theorem 4.8, tondji2024acceleration
  • ...and 8 more