Reflective block Kaczmarz algorithms for least squares

TL;DR

This work rigorously develops reflective Kaczmarz algorithms for least squares, including randomised, block, and deterministic variants. It establishes convergence rates for both consistent and inconsistent cases, with key metrics such as the scaled condition number $\kappa_F$ and block parameters governing speedups. A central contribution is the detailed analysis of the geometric structure, notably the sphere-centre interpretation and the role of the quantity $\eta(A)$ in the deterministic setting, along with new connections to a modified LS problem via $A^T L^{-1}A$ and Coxeter-type eigenvalue characterisations. The results supply both theoretical guarantees and practical guidance, supported by numerical experiments that confirm faster convergence with block updates and the utility of restarting strategies across settings. This advances understanding of Kaczmarz-type methods by linking LS solutions to geometric centers and by providing robust convergence theory for both randomized and deterministic reflective variants, including their block generalisations.

Abstract

In [Steinerberger, Q. Appl. Math., 79:3, 419-429, 2021] and [Shao, SIAM J. Matrix Anal. Appl. 44(1), 212-239, 2023], two new types of Kaczmarz algorithms, which share some similarities, for consistent linear systems were proposed. These two algorithms not only compete with many previous Kaczmarz algorithms but, more importantly, reveal some interesting new geometric properties of solutions to linear systems that are not obvious from the standard viewpoint of the Kaczmarz algorithm. In this paper, we comprehensively study these two algorithms. First, we theoretically analyse the algorithms given in [Steinerberger, Q. Appl. Math., 79:3, 419-429, 2021] for solving least squares. Second, we extend the two algorithms to block versions and provide their theoretical convergence rates. Our numerical experiments also verify the efficiency of these algorithms. Third, as a theoretical complement, we address some key questions left unanswered in [Shao, SIAM J. Matrix Anal. Appl. 44(1), 212-239, 2023].

Figures (5)

• Figure 1: Comparison of reflective Kaczmarz algorithm (\ref{['standard verison']}) for consistent and inconsistent linear systems. The red point in the center is the optimal solution. The blue points are generated by the procedure (\ref{['standard verison']}).
• Figure 2: Comparison of Algorithm 1 and Algorithm 2.
• Figure 3: Comparison of Algorithm 2 for different choices of $q$ for random linear systems of size $300\times 100$.
• Figure 4: The runtime (in seconds) of Algorithm 2 for different choices of $q\in\{1,2,\ldots,100\}$ for random linear systems of size $300\times 100$.
• Figure 5: A demonstration of ${\bf x}_*^{(i)}$, which is denoted as $s_i$ because of some reason of exhibition, and ${\bf x}_{\rm LS}$ in dimension 2.

Theorems & Definitions (67)

• Proposition 2.1: see stewart2004elsner
• Remark 2.2
• Proposition 2.3: Theorem 2.1 in elsner1982perturbation
• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Theorem 3.3: Convergence rate of reflective Kaczmarz algorithm for least squares
• proof
• Definition 4.1: Block Householder matrix