Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kaczmarz Kac Walk

Stefan Steinerberger

TL;DR

A sequence of linear systems is proposed which is fast to compute, preserves the original solution and whose small singular values grow like $\sigma_{\ell}(A^{(k)}) \sim \exp(k/n^2) \cdot \sigma_{\ell}(A)$.

Abstract

The Kaczmarz method is a way to iteratively solve a linear system of equations $Ax = b$. One interprets the solution $x$ as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems $A^{(k)} x = b^{(k)}$ which is fast to compute, preserves the original solution and whose small singular values grow like $σ_{\ell}(A^{(k)}) \sim \exp(k/n^2) \cdot σ_{\ell}(A)$.

Kaczmarz Kac Walk

TL;DR

A sequence of linear systems is proposed which is fast to compute, preserves the original solution and whose small singular values grow like .

Abstract

The Kaczmarz method is a way to iteratively solve a linear system of equations . One interprets the solution as the point where hyperplanes intersect and then iteratively projects an approximate solution onto these hyperplanes to get better and better approximations. We note a somewhat related idea: one could take two random hyperplanes and project one into the orthogonal complement of the other. This leads to a sequence of linear systems which is fast to compute, preserves the original solution and whose small singular values grow like .

Paper Structure

This paper contains 10 sections, 2 theorems, 42 equations, 6 figures.

Table of Contents

  1. Introduction and Result
  2. The Kaczmarz method.
  3. Kaczmarz Kac Walk
  4. Main Result
  5. The overdetermined case.
  6. Kac walk.
  7. Proof of the Theorem
  8. Proof of the Theorem
  9. Remark.
  10. The case $A \in \mathbb{R}^{n \times 2}$

Key Result

Theorem 1

If the projection onto hyperplane $H_i$ is chosen with likelihood $\|A_i\|^2/\|A\|_F^2$, then where $\sigma_{\min}$ is the smallest singular value of $A$.

Figures (6)

  • Figure 1: Projection $\pi_i y$ onto $H_i$ given by $\left\langle a_i, w\right\rangle = b_i$.
  • Figure 2: The smallest singular value of random Gaussian $100 \times 100$ matrix over 20000 (left) and 80000 (right) iterations.
  • Figure 3: Prediction \ref{['pred']} in red tested against ten random evolutions of $\sigma_{100}(A^{(k)})$ for a random initial matrix of size $100 \times 100$. We observe high accuracy while $\sigma_{100}(A^{(k)}) \ll 1$.
  • Figure 4: Prediction \ref{['pred3']} tested against ten random evolutions of $\sigma_{100}(A^{(k)})$ (left) and $\sigma_{70}(A^{(k)}$ (right) for a random $100 \times 100$ matrix.
  • Figure 5: Evolution of a $100 \times 25$ matrix: $\sigma_1(A^{(k)})/\sigma_{25}(A^{(k)})$ (left) and empirical density of all singular values (right).
  • ...and 1 more figures

Theorems & Definitions (3)

  • Theorem : Strohmer--Vershynin
  • Theorem
  • proof