Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $λ$-Effectiveness

Halyun Jeong, Palle E. T. Jorgensen, Hyun-Kyoung Kwon, Myung-Sin Song

TL;DR

The work develops an operator-theoretic framework for regret analysis of block Kaczmarz algorithms in infinite-dimensional Hilbert spaces, proving a dimension-free $O(1/k)$ average regret for generalized updates with relaxation $\lambda\in(0,2)$. It separates estimation error from a noise floor in the presence of additive noise, yielding a clear guidance on the choice of $\lambda$ under noise and horizon length. A central contribution is the $\,\lambda$-effectiveness theory for Fourier exponential systems, showing that $\{e^{2\pi i n x}\}_{n\ge0}$ is $\lambda$-effective for all $\lambda\in(0,2)$ if and only if the underlying spectral measure is singular (with $\lambda=1$ recovering the classical Lebesgue case). The authors develop Hardy-space and inner-function tools to derive a $\lambda$-dependent inner-function criterion, construct generalized Kaczmarz-Fourier expansions, and connect these with a generalized Cauchy transform, enabling robust non-orthogonal harmonic-analysis representations and applications to signal processing and infinite-dimensional online learning. Overall, the paper unifies regret bounds, spectral-measure analysis, and Fourier-analytic expansions within an operator-theoretic framework for projection-based learning in infinite-dimensional settings.

Abstract

We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit $λ$-dependence to quantify how much an algorithm's performance deviates from its optimal performance. A detailed analysis of relaxation parameter is also provided. Applications include: explicit regret bounds for the framework of Kaczmarz algorithm models, non-orthogonal Fourier expansions, and the use of regret estimates in modern machine learning models, including for noisy data, i.e., regret bounds for the noisy Kaczmarz algorithms. Motivated by machine-learning practice, our wider framework treats bounded operators (on infinite-dimensional Hilbert spaces), with updates realized as (block) Kaczmarz algorithms, leading to new and versatile results.

Infinite-Dimensional Operator/Block Kaczmarz Algorithms: Regret Bounds and $λ$-Effectiveness

TL;DR

The work develops an operator-theoretic framework for regret analysis of block Kaczmarz algorithms in infinite-dimensional Hilbert spaces, proving a dimension-free average regret for generalized updates with relaxation . It separates estimation error from a noise floor in the presence of additive noise, yielding a clear guidance on the choice of under noise and horizon length. A central contribution is the -effectiveness theory for Fourier exponential systems, showing that is -effective for all if and only if the underlying spectral measure is singular (with recovering the classical Lebesgue case). The authors develop Hardy-space and inner-function tools to derive a -dependent inner-function criterion, construct generalized Kaczmarz-Fourier expansions, and connect these with a generalized Cauchy transform, enabling robust non-orthogonal harmonic-analysis representations and applications to signal processing and infinite-dimensional online learning. Overall, the paper unifies regret bounds, spectral-measure analysis, and Fourier-analytic expansions within an operator-theoretic framework for projection-based learning in infinite-dimensional settings.

Abstract

We present a variety of projection-based linear regression algorithms with a focus on modern machine-learning models and their algorithmic performance. We study the role of the relaxation parameter in generalized Kaczmarz algorithms and establish a priori regret bounds with explicit -dependence to quantify how much an algorithm's performance deviates from its optimal performance. A detailed analysis of relaxation parameter is also provided. Applications include: explicit regret bounds for the framework of Kaczmarz algorithm models, non-orthogonal Fourier expansions, and the use of regret estimates in modern machine learning models, including for noisy data, i.e., regret bounds for the noisy Kaczmarz algorithms. Motivated by machine-learning practice, our wider framework treats bounded operators (on infinite-dimensional Hilbert spaces), with updates realized as (block) Kaczmarz algorithms, leading to new and versatile results.

Paper Structure

This paper contains 42 sections, 20 theorems, 257 equations.

Table of Contents

  1. Introduction
  2. Organization
  3. The Average Regret Bound of the Kaczmarz Algorithm
  4. Regret Bound for Kaczmarz Algorithm with Noisy Observations
  5. Generalized Kaczmarz with Step Size lambda in (0,2)
  6. Preliminaries on the Generalized Kaczmarz operators
  7. Proof of (i)
  8. Proof of (ii)
  9. Noisy Kaczmarz regret bound with step size lambda in (0,2)
  10. Generalized Kaczmarz Algorithm and $\lambda$-Effectiveness
  11. Orthogonal case.
  12. Non-Orthogonal case.
  13. Auxiliary Sequence of the Generalized Kaczmarz Algorithm
  14. Equivalence of recursive and combinatorial definitions of $\alpha_n^{(\lambda)}$
  15. Relation to combinatorial coefficients
  16. ...and 27 more sections

Key Result

Theorem 2.4

Let $\{w_t\}_{t=0}^k$ be the sequence of updates by the (block) Kaczmarz algorithm with the initial condition $w_0 = 0$. Suppose there is a parameter vector $w^*$ such that $y_t = X_tw^*$ for all $t = 1, 2, \dots, k$. Suppose that Assumption asmp:task_bound holds with $\|X_t\| \le 1$ and Assumption which shows it is bounded by $O(1/k)$. If we assume further that the system $\{ P_t \}_{t \ge 0}$ i

Theorems & Definitions (46)

  • Definition 2.3: Effectiveness and $\lambda$--effectiveness
  • Theorem 2.4
  • Remark 2.5: Rank-one partial isometries
  • proof : Proof of Theorem \ref{['thm:kaczmarz']}
  • Theorem 3.1
  • Remark 3.2
  • Remark 3.3
  • proof
  • Lemma 4.1
  • proof
  • ...and 36 more