On greedy multi-step inertial randomized Kaczmarz method for solving linear systems

TL;DR

It is proved that the proposed greedy MIRK (GMIRK) method enjoys an improved deterministic linear convergence compared to both the MIRK method and the greedy randomized Kaczmarz method.

Abstract

The multi-step inertial randomized Kaczmarz (MIRK) method is an iterative method for solving large-scale linear systems. In this paper, we enhance the MIRK method by incorporating the greedy probability criterion, coupled with the introduction of a tighter threshold parameter for this criterion. We prove that the proposed greedy MIRK (GMIRK) method enjoys an improved deterministic linear convergence compared to both the MIRK method and the greedy randomized Kaczmarz method. Furthermore, we exhibit that the multi-step inertial extrapolation approach can be geometrically interpreted as an orthogonal projection method, and establish its relationship with the sketch-and-project method in (SIAM J. Matrix Anal. Appl. 36(4):1660-1690, 2015) and the oblique projection technique in (Results Appl. Math. 16:100342, 2022). Numerical experiments are provided to confirm our results.

Figures (7)

• Figure 1: A geometric interpretation of the multi-step inertial extrapolation approach. The next iterate $x^{(k+1)}$ is the projection of $w^{(k)}=x^{(k)}+\beta_k a_{i_{k-1}}$ onto the hyperplane $H_{i_k}$ with $\beta_k$ being chosen such that $x^{(k+1)}$ locates on the intersection of the two hyperplanes $H_{i_k}$ and $H_{i_{k-1}}$.
• Figure 2: The next iterate $x^{(k+1)}$ is the only point in the intersection of $H_{i_k} \cap H_{i_{k-1}}$ and $\Pi_k$. It is the projection of $x^\star$ onto $\Pi_k$ and meanwhile the projection of $x^{(k)}$ onto $H_{i_k} \cap H_{i_{k-1}}$.
• Figure 3: A visualization of a typical Kaczmarz iteration that projects the iterates back-and-forth between the two hyperplanes $H_{i_k}$ and $H_{i_{k-1}}$.
• Figure 4: A geometric explanation of the oblique projection technique. The next iterate $x^{(k+1)}$ is the projection of $x^{(k)}$ along the direction $d^{(k)}$ onto the hyperplane $H_{i_k}$.
• Figure 5: Figures depict the value of $\delta_{\min}$ vs increasing number of rows ($n=500$) or columns ($m=500$). We set $t=0.1,0.5$, and $0.9$, respectively.

Theorems & Definitions (11)

• Remark 2.1
• Lemma 2.2
• proof
• Lemma 2.3
• proof
• Theorem 2.4
• proof
• Remark 2.5
• Remark 2.6
• Remark 2.7