Where Have All the Kaczmarz Iterates Gone?

TL;DR

This work analyzes the randomized Kaczmarz (RK) algorithm on noisy and potentially inconsistent linear systems, extending prior results to arbitrary reference points and removing stringent initialization assumptions. It derives convergence bounds for the RK iterates in expectation and reveals how singular-vector structure of the noisy matrix governs convergence directions, including explicit expressions along right and left singular vectors. The authors show that the RK iterates asymptotically cluster to a ball with a radius (convergence horizon) determined by noise via terms like $\|Ex_{LS}-\epsilon\|$ and the smallest singular value $\tilde{\sigma}_{\min}$, and that the smallest horizon is achieved when the reference aligns with the noisy LS solution $\widetilde{x}_{LS}$ (up to a null-space component). Numerical results on synthetic and real data corroborate the bounds and demonstrate the practical implications for RK robustness, highlighting that in general one can expect proximity to the noisy LS solution rather than the original clean solution unless special noise structures are present.

Abstract

The randomized Kaczmarz (RK) algorithm is one of the most computationally and memory-efficient iterative algorithms for solving large-scale linear systems. However, practical applications often involve noisy and potentially inconsistent systems. While the convergence of RK is well understood for consistent systems, the study of RK on noisy, inconsistent linear systems is limited. This paper investigates the asymptotic behavior of RK iterates in expectation when solving noisy and inconsistent systems, addressing the locations of their limit points. We explore the roles of singular vectors of the (noisy) coefficient matrix and derive bounds on the convergence horizon, which depend on the noise levels and system characteristics. Finally, we provide extensive numerical experiments that validate our theoretical findings, offering practical insights into the algorithm's performance under realistic conditions. These results establish a deeper understanding of the RK algorithm's limitations and robustness in noisy environments, paving the way for optimized applications in real-world scientific and engineering problems.

Figures (9)

• Figure 1: Path of the RK iterates. RK is applied to $\widetilde{A}x \approx \widetilde{b}$ for $\widetilde{A} \in \mathbb{R}^{3\times 2}$. We consider the cases in which (left) ${\rm rank}(\widetilde{A})=2$ and $x_0,x^1,x^2,x^3 \in {\rm range}(\widetilde{A} ^\top)$ and (right) ${\rm rank}(\widetilde{A})=1$ and $x_0,x^1,x^2,x^3 \notin {\rm range}(\widetilde{A} ^\top)$. Each line represents a solution space determined by a row of the linear equalities $\widetilde{A}x \approx \widetilde{b}$. The point $\widetilde{x}_{\rm LS}=\widetilde{A}^\dagger \widetilde{b}$ and $x^1,x^2,x^3$ are arbitrary. The circles are of centers $x_0^n+x_*^r$ and radius $\|\widetilde{A} \widetilde{x}_* - \widetilde{b} \|/\widetilde{\sigma}_{min}$ for $x_*=\widetilde{x}_{\rm LS},x^1,x^2,x^3$. The intersection of all the circles is going to be where all the cluster points are.
• Figure 2: Path of RK iterates. The system, $Ax=b$ is consistent with $A \in \mathbb{R}^{6 \times 3}$ of rank $2$ and $x_{\rm LS}=A^\dagger b$. (a) RK applied to $Ax = b$ with $x_0 \notin {\rm Range} ({A}^\top)$. (b) RK applied to $\tilde{A}x\approx \tilde{b}$ with $\tilde{A} = A+E$, $\tilde{b} = b+\epsilon$, and $x_0 - x_{\rm LS} \notin {\rm Range} ( \tilde{A}^\top)$. The radius of the ball centered around $x_{\rm LS}$ is $\|Ex_{\rm LS}-\epsilon\|/{\tilde{\sigma}_{\rm min}}$.
• Figure 3: The approximation errors $\sqrt{{\mathbb{E}}\left[\|x_k-x_0^n-x_*^r\|^2\right] }$ and $\|{\mathbb{E}}\left[x_k\right] -x_0^n-x_*^r\|$ of RK applied to $\widetilde{A}x \approx \widetilde{b}$, square root of bound of Theorem \ref{['thm:DoublyNoisy_x']}, and bound of Corollary \ref{['cor:better estimate']}. We have $m=1000$, $n=500$, and ${\rm rank}(\widetilde{A})=300$. The initialization $x_0$ and reference point $x_*$ are random. We have $\|\widetilde{b}_{{\rm Col}(\widetilde{A})^\perp}\|=\beta$. On the left, $\beta=10$. Here, on the right, $\beta=10000$.
• Figure 4: The approximation errors $\sqrt{{\mathbb{E}}\left[\|x_k-x_0^n-\widetilde{x}_{\rm LS}\|^2\right] }$ and $\|{\mathbb{E}}\left[x_k\right] -x_0^n- \widetilde{x}_{\rm LS}\|$ of RK applied to $\widetilde{A}x \approx \widetilde{b}$, square root of bound of Theorem \ref{['thm:DoublyNoisy_x']}, and bound of Corollary \ref{['cor:better estimate']}. We have $m=1000$, $n=500$, and ${\rm rank}(\widetilde{A})=300$. The initialization $x_0$ is random and $\widetilde{x}_{\rm LS}=\widetilde{A}^\dagger\widetilde{b}$. We have $\|\widetilde{b}_{{\rm Col}(\widetilde{A})^\perp}\|=\beta$. On the left, $\beta=10$. On the right, $\beta=10000$.
• Figure 5: The approximation errors $\sqrt{{\mathbb{E}}\left[\|x_k-x_0^n-\widetilde{v}_{\rho}\|^2\right] }$ and $\|{\mathbb{E}}\left[x_k\right] -x_0^n- \widetilde{v}_{\rho}\|$ of RK applied to $\widetilde{A}x=0$$(\widetilde{b}=0)$ with ${\rm rank}(\widetilde{A})=\rho$, square root of bound of Theorem \ref{['thm:DoublyNoisy_x']}, and bound of Corollary \ref{['cor:better estimate']}. The initialization $x_0$ is arbitrary. On the left, $\widetilde{A}$ is of low rank. On the right, $\widetilde{A}$ is full-rank.

Theorems & Definitions (19)

• Theorem 1: Strohmer2013, Theorem 2
• Theorem 2: zouzias2013randomized, Theorem 2.1
• Theorem 3: doubly_noisy_RK, Theorem 3.1
• Theorem 4
• Theorem 5
• proof
• Theorem 6: steinerberger2021randomized, Theorem 1
• Theorem 7
• proof
• Remark 1