A Kaczmarz-Inspired Method for Orthogonalization
Rikhav Shah, Isabel Detherage
TL;DR
The paper investigates a randomized, Kaczmarz-inspired procedure to orthogonalize a set of unit vectors while preserving their span. It analyzes the evolution of the n-volume det(|A|) of the parallelepiped formed by the vectors through the potential Φ(A) = -\log det(|A|), proving convergence to an orthonormal basis of Col(A) and providing quantitative rates: after t = Θ(n^2 log(1/(det(|A|) ε)) steps, det(|A_t|) exceeds 1-ε with constant probability. The analysis combines exact one-step determinant updates, a convex-analytic bound f, and a supermartingale framework to obtain expectation and tail bounds on Φ(A_t) and related convergence metrics. The work also connects this approach to QR/Gram-Schmidt, highlights algorithmic advantages such as early stopping for conditioning and parallelizability, and suggests directions for improving rates and extending to batch updates.
Abstract
This paper asks if the following iterative procedure approximately orthogonalizes a set of $n$ linearly independent unit vectors while preserving their span: in each iteration, access a random pair of vectors and replace one with the component perpendicular to the other, renormalized to be a unit vector. We provide a positive answer: any given set of starting vectors converges almost surely to an orthonormal basis of their span. We specifically argue that the $n$-volume of the parallelepiped generated by the vectors approaches 1 (i.e. the parallelepiped approaches a hypercube). If $A$ is the matrix formed by taking these vectors as columns, this volume is simply $\det(|A|)$ where $|A|=(A^*A)^{1/2}$. We show that $O(n^2\log(1/(\det(|A|)\varepsilon)))$ iterations suffice to bring ${\det(|A|)}$ above $1-\varepsilon$ with constant probability.