Survey of a Class of Iterative Row-Action Methods: The Kaczmarz Method

TL;DR

This survey analyzes the Kaczmarz family of row-action iterative methods for solving linear systems, benchmarking the original cyclic version against a broad spectrum of randomized and block variants. It highlights how row/column sampling strategies, including uniform, norm-based, greedy, and quasirandom schemes, influence convergence on large, dense problems and in CT-reconstruction contexts. The work shows that certain variants (e.g., SRKWOR, SRK-Halton/Sobol) can outperform the classical RK and CK in consistent systems, while in inconsistent cases, Conjugate Gradient for Least Squares (CGLS) often remains the practical choice. The authors also discuss parallel implementations and CT-specific adaptations, emphasizing semi-convergence and stopping criteria in the presence of noise. Overall, the study provides a comprehensive framework for selecting Kaczmarz-type solvers based on problem structure, data characteristics, and computational resources, with implications for real-time and large-scale data applications.

Abstract

The Kaczmarz algorithm is an iterative method that solves linear systems of equations. It stands out among iterative algorithms when dealing with large systems for two reasons. First, at each iteration, the Kaczmarz algorithm uses a single equation, resulting in minimal computational work per iteration. Second, solving the entire system may only require the use of a small subset of the equations. These characteristics have attracted significant attention to the Kaczmarz algorithm. Researchers have observed that randomly choosing equations can improve the convergence rate of the algorithm. This insight led to the development of the Randomized Kaczmarz algorithm and, subsequently, several other variations emerged. In this paper, we extensively analyze the native Kaczmarz algorithm and many of its variations using large-scale dense random systems as benchmarks. Through our investigation, we have verified that, for consistent systems, various row sampling schemes can outperform both the original and Randomized Kaczmarz method. Specifically, sampling without replacement and using quasirandom numbers are the fastest techniques. However, for inconsistent systems, the Conjugate Gradient method for Least-Squares problems overcomes all variations of the Kaczmarz method for these types of systems.

Figures (12)

• Figure 1: Convergence of the Kaczmarz method in 2 dimensions for consistent and inconsistent systems.
• Figure 2: Convergence of the Kaczmarz method for a consistent system in 2 dimensions using two different row selection criteria.
• Figure 3: Distribution of 50 sampled numbers in interval $\left[ 1, 1000 \right]$.
• Figure 4: Spatial domain of a CT scan. The body that is being scanned is represented in red and the projections of each ray are represented in blue. $\theta$ is the rotation angle of the source.
• Figure 5: Application on an iterative method to solve problems derived from CT scans. The image shows the evolution of the reconstruction error during several iterations.