On spectral properties and fast initial convergence of the Kaczmarz method

TL;DR

The paper addresses why the Kaczmarz method can exhibit rapid initial convergence yet slow asymptotic convergence for discrete ill-posed problems, particularly in CT. It develops a spectral analysis of the nonnormal iteration operator $G$ restricted to the invariant subspace $\mathcal{V} = \mathrm{range}(A^T)$ and a statistical model of noise propagation to explain semi-convergence. Key findings show that near-zero eigenvalues of $G$ (often arising from row structure and $oldsymbol{}=1$) drive fast early progress, while upper bounds for $\rho(G|_{\\mathcal{V}})$ quantify slow long-term convergence; the small-$\omega$ regime reveals quasi-symmetric behavior that can balance speed and stability. These insights explain practical CT performance and offer guidance on row ordering and relaxation parameter selection to manage early progress, asymptotic behavior, and noise amplification in inverse problems.

Abstract

The Kaczmarz method is successfully used for solving discretizations of linear inverse problems, especially in computed tomography where it is known as ART. Practitioners often observe and appreciate its fast convergence in the first few iterations, leading to the same favorable semi-convergence that we observe for simultaneous iterative reconstruction methods. While the latter methods have symmetric and positive definite iteration operators that facilitate their analysis, the operator in Kaczmarz's method is nonsymmetric and it has been an open question so far to understand this fast initial convergence. We perform a spectral analysis of Kaczmarz's method that gives new insight into its (often fast) initial behavior. We also carry out a statistical analysis of how the data noise enters the iteration vectors, which sheds new light on the semi-convergence. Our results are illustrated with several numerical examples.

Figures (11)

• Figure 1: A typical distribution of the eigenvalues of $G$ inside the complex unit disk for an X-ray CT problem. The spectral radius here is $\rho(G) = 0.9999998937$.
• Figure 2: CT test problem (see text for details). Left: the small eigenvalues of $G$ and $\widetilde{G}$ corresponding, respectively, to the matrices $A$ and $\widetilde{A}$ with default and random row ordering. Middle: the nonzeros of the leading submatrices of $AA^\top$ and $\widetilde{A}\widetilde{A}^\top$. Right: the error histories for the two row orderings.
• Figure 3: Problems gravity (${\mathtt d} = 0.01$ and $n=128$) and paralleltomo (the image size is $32\times 32$, and $A$ is $1024 \times 1024$), with, respectively, one zero eigenvalue, and several zero eigenvalues for an interval around $\omega=1$.
• Figure 4: Convergence of the Kaczmarz methods for the gravity test problem with ${\mathtt d}=0.06$ and $n=128$. The inset plot shows the absolute values of the elements in $\mathbf y = W^{-1} \mathbf x_{\infty}$, i.e., the coefficients of the solution in the eigenvector basis. The fast initial convergence is explained by the fast decay of $|y_i|$.
• Figure 5: Comparison of the Kaczmarz and symmetric Kaczmarz methods for the gravity test problem with d = 0.01 (left), 0.02 (middle), and 0.4 (right). In all three plots, we see an initial fast convergence followed by a phase with slow asymptotic convergence.

Theorems & Definitions (25)

• Lemma 1
• proof
• Proposition 2
• proof
• Proposition 3
• proof
• Example 4
• Proposition 5
• proof
• Example 6