A Note on Randomized Kaczmarz Algorithm for Solving Doubly-Noisy Linear Systems

TL;DR

This paper analyzes the convergence of RK foroubly-noisy linear systems, i.e., when the coefficient matrix, $A$ has additive or multiplicative noise, and $b$ is also noisy, and claims that the analysis is robust and realistically applicable.

Abstract

Large-scale linear systems, $Ax=b$, frequently arise in practice and demand effective iterative solvers. Often, these systems are noisy due to operational errors or faulty data-collection processes. In the past decade, the randomized Kaczmarz (RK) algorithm has been studied extensively as an efficient iterative solver for such systems. However, the convergence study of RK in the noisy regime is limited and considers measurement noise in the right-hand side vector, $b$. Unfortunately, in practice, that is not always the case; the coefficient matrix $A$ can also be noisy. In this paper, we analyze the convergence of RK for {\textit{doubly-noisy} linear systems, i.e., when the coefficient matrix, $A$, has additive or multiplicative noise, and $b$ is also noisy}. In our analyses, the quantity $\tilde R=\| \tilde A^{\dagger} \|^2 \|\tilde A \|_F^2$ influences the convergence of RK, where $\tilde A$ represents a noisy version of $A$. We claim that our analysis is robust and realistically applicable, as we do not require information about the noiseless coefficient matrix, $A$, and considering different conditions on noise, we can control the convergence of RK. {We perform numerical experiments to substantiate our theoretical findings.}

Figures (5)

• Figure 1: Approximation error, $\|x_k - x_{{\rm LS}}\|$, and the theoretical bounds of Theorems \ref{['theorem:ls_rk_pnls']} and \ref{['theorem:general_theorem']} for RK on a noisy linear system. $Ax=b$ and $\Tilde{A}x=b$ are consistent with $m=500$, $n=300$, $r=300$, $\kappa_A=10$, $\sigma_{r}(A)=5$, and $\sigma_{1}(A)=50$. The entries of $\epsilon$ are generated from the standard Normal distribution.
• Figure 2: Approximation error, $\|x_k - x_{{\rm LS}}\|^2$, and the theoretical bound of Corollary \ref{['theorem:general_theorem_multiplicative']} for RK on a noisy linear system. $Ax=b$ is consistent with $m=500$, $n=300$, $r=300$, $\kappa_A=10$, $\sigma_{r}(A)=1$, and $\sigma_{1}(A)=10$. The entries of $E$ and $\epsilon$ are generated from the standard normal distribution, $F=0$.
• Figure 3: Approximation error, $\|x_k - x_{{\rm LS}}\|^2$, and the theoretical bound of Corollary \ref{['theorem:general_theorem_multiplicative']} for RK on a noisy linear system. $Ax=b$ is consistent with $m=500$, $n=300$, $r=300$, $\kappa_A=10$, $\sigma_{r}(A)=1$, and $\sigma_{1}(A)=10$. The entries of $F$ and $\epsilon$ are generated from the standard Normal distribution, and $E=0$.
• Figure 4: Approximation error, $\|x_k - x_{{\rm LS}}\|^2$, and the theoretical bound of Corollary \ref{['theorem:general_theorem_multiplicative']} for RK on a noisy linear system. $Ax=b$ is consistent with $m=500$, $n=300$, $r=300$, $\kappa_A=10$, $\sigma_{r}(A)=1$, and $\sigma_{1}(A)=10$. The entries of $F$, $E$, and $\epsilon$ are generated from the standard Normal distribution.
• Figure 5: Approximation error, $\|x_k - x_{{\rm LS}}\|^2$, for RK on noise-free system $Ax=b$, and on noisy system $\Tilde{A}x=b$. On the left: $m=100$, $n=50$, $r=50$, $R=785$ and $\Tilde{R}=50$. On the right $m=100$, $n=100$, $r=100$, $R=1585$ and $\Tilde{R}=100$. For both cases $\sigma_{r}(A)=1$, and $\sigma_{i}(A)=4$ for $i=1,...,r-1$.

Theorems & Definitions (14)

• Theorem 2.1
• Lemma 2.2
• Theorem 2.3
• Proof 1
• Theorem 2.4
• Proof 2
• Theorem 3.1
• Proof 3
• Remark 3.2
• Proposition 3.3