Randomly sparsified Richardson iteration: A dimension-independent sparse linear solver

TL;DR

This work introduces randomly sparsified Richardson iteration (RSRI), a dimension-independent sparse linear solver that combines Richardson fixed-point iterations with random sparsification to solve $A x=b$ using at most $m$ nonzeros per update. The authors provide a complete mathematical analysis showing faster-than-Monte Carlo convergence, with explicit bias-variance bounds and conditions under which RSRI converges, even when the solution vector is too large to store densely. The PageRank problem is used as a primary application, where RSRI achieves improved variance decay and demonstrates practical scalability to very large graphs, aided by the pivotal sparsification design. By situating RSRI relative to Monte Carlo methods, SGD, deterministic sparse methods, and fast randomized iteration, the paper establishes a principled framework for sparsified fixed-point solvers with provable performance, including near-optimal triple-norm error behavior and explicit guidelines for choosing sparsity $m$.

Abstract

Recently, a class of algorithms combining classical fixed point iterations with repeated random sparsification of approximate solution vectors has been successfully applied to eigenproblems with matrices as large as $10^{108} \times 10^{108}$. So far, a complete mathematical explanation for their success has proven elusive. The family of methods has not yet been extended to the important case of linear system solves. In this paper we propose a new scheme based on repeated random sparsification that is capable of solving sparse linear systems in arbitrarily high dimensions. We provide a complete mathematical analysis of this new algorithm. Our analysis establishes a faster-than-Monte Carlo convergence rate and justifies use of the scheme even when the solution vector itself is too large to store.

Figures (2)

• Figure 1: (RSRI error scaling.) Left panel shows tail decay $\sum\nolimits_{i = m}^n \bm{x}^{\downarrow}(i)$ of the sorted PageRank solution $\bm{x}^{\downarrow}$ for three personalized PageRank problems documented in \ref{['sec:empirical']}. Right panel shows RSRI error $(\mathbb{E} \lVert \hat{\bm{x}} - \bm{x}_\star \rVert^2)^{1/2}$ with sparsity level $m$. The right panel is essentially the left panel multiplied by a factor of $m^{-1/2}$
• Figure 1: (Slow convergence of coordinate descent). Error $\lVert \hat{\bm{x}} - \bm{x} \rVert$ and residual error $\lVert \bm{r}(\hat{\bm{x}}) \rVert_{\infty}$ for coordinate descent with $t$ update steps, compared to theoretical $t^{-1}$ scaling. In the figure, we apply coordinate descent to the Amazon electronics PageRank problem, as documented in \ref{['sec:empirical']}.

Theorems & Definitions (27)

• Theorem 2.1: Main error bound
• Corollary 2.2: Fast polynomial or exponential convergence
• Proof 1
• Proposition 3.1: PageRank error bound
• Lemma 3.2: Decay of entries in the PageRank vector
• Proof 2: Proof of \ref{['lem:decay_rate']}
• Theorem 5.1: Near-optimal error
• Proposition 5.2: Optimal $L^2$ error
• Proof 3: Proof of \ref{['prop:relaxed']}
• Lemma 5.3: Difference in norms