Randomized Kaczmarz with tail averaging

TL;DR

This work introduces tail-averaged randomized Kaczmarz (TARK) to overcome the finite-horizon limitation of RK in inconsistent least-squares problems, achieving convergence to the exact solution with a hybrid exponential-plus-polynomial rate that decomposes into bias and variance terms. By averaging RK iterates over a tail after a burn-in, TARK attains the optimal $O(1/t)$ rate for row-access methods while maintaining a practical, storage-efficient implementation; the analysis is supported by a Demmel-condition-based contraction framework and multi-step expectations. The paper also extends the approach to semi-infinite and ridge-regularized problems (TARK-RR), providing explicit bounds and demonstrating the benefits of regularization and tail averaging through numerical experiments. Overall, the results advance scalable, provably convergent row-access solvers for large-scale least-squares and regularized variants, with implications for preconditioning and block-augmented strategies.

Abstract

The randomized Kaczmarz (RK) method is a well-known approach for solving linear least-squares problems with a large number of rows. RK accesses and processes just one row at a time, leading to exponentially fast convergence for consistent linear systems. However, RK fails to converge to the least-squares solution for inconsistent systems. This work presents a simple fix: average the RK iterates produced in the tail part of the algorithm. The proposed tail-averaged randomized Kaczmarz (TARK) converges for both consistent and inconsistent least-squares problems at a polynomial rate, which is known to be optimal for any row-access method. An extension of TARK also leads to efficient solutions for ridge-regularized least-squares problems.

Figures (2)

• Figure 1: Left: Relative errors for four RK methods from Table \ref{['tab:comparison']} on a polynomial regression task. Right: Computed polynomials for RK and TARK compared to target function.
• Figure 2: Relative errors for RK methods applied to un-regularized (left) and regularized (right) polynomial regression problems.

Theorems & Definitions (24)

• Theorem 1: Randomized Kaczmarz: Convergence to a horizon zouzias2013randomized
• Theorem 2: Mean square error bound for TARK
• Theorem 3: Alternative TARK error bound
• Lemma 1: Multi-step expectations
• proof
• Lemma 2: Demmel condition number bound
• proof
• Lemma 3: Mean square errors, based on strohmer2008randomized
• proof
• proof : Proof of Theorems \ref{['thm:variance_simple']} and \ref{['thm:alternative']}