Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Randomized block Kaczmarz with volume sampling: Momentum acceleration and efficient implementation

Ruike Xiang, Jiaxin Xie, Qiye Zhang

TL;DR

The paper addresses solving $A x=b$ by enhancing randomized block Kaczmarz with volume sampling (RBKVS) and momentum acceleration. It introduces volume sampling where the probability of selecting a row-subset is proportional to the volume of the corresponding submatrix $A_{\mathcal{S}}A_{\mathcal{S}}^{\top}$, and extends RBK with heavy-ball momentum to achieve faster convergence. The authors prove linear convergence bounds for RBKVS and accelerated rates for the momentum variant, and they propose an efficient sparse-volume-sampling implementation with preprocessing costs proportional to the number of nonzeros and a log-time sampling step. Numerical experiments on average-consensus problems and SuiteSparse matrices validate the theory and show substantial gains in both iterations and runtime, demonstrating the practicality of the approach for large-scale sparse systems.

Abstract

The randomized block Kaczmarz (RBK) method is a widely utilized iterative scheme for solving large-scale linear systems. However, the theoretical analysis and practical effectiveness of this method heavily rely on a good row paving of the coefficient matrix. This motivates us to introduce a novel block selection strategy to the RBK method, called volume sampling, in which the probability of selection is proportional to the volume spanned by the rows of the selected submatrix. To further enhance the practical performance, we develop and analyze a momentum variant of the method. Convergence results are established and demonstrate the notable improvements in convergence factor of the RBK method brought by the volume sampling and the momentum acceleration. Furthermore, to efficiently implement the RBK method with volume sampling, we propose an efficient algorithm that enables volume sampling from a sparse matrix with sampling complexity that is only logarithmic in dimension. Numerical experiments confirm our theoretical results.

Randomized block Kaczmarz with volume sampling: Momentum acceleration and efficient implementation

TL;DR

The paper addresses solving by enhancing randomized block Kaczmarz with volume sampling (RBKVS) and momentum acceleration. It introduces volume sampling where the probability of selecting a row-subset is proportional to the volume of the corresponding submatrix , and extends RBK with heavy-ball momentum to achieve faster convergence. The authors prove linear convergence bounds for RBKVS and accelerated rates for the momentum variant, and they propose an efficient sparse-volume-sampling implementation with preprocessing costs proportional to the number of nonzeros and a log-time sampling step. Numerical experiments on average-consensus problems and SuiteSparse matrices validate the theory and show substantial gains in both iterations and runtime, demonstrating the practicality of the approach for large-scale sparse systems.

Abstract

The randomized block Kaczmarz (RBK) method is a widely utilized iterative scheme for solving large-scale linear systems. However, the theoretical analysis and practical effectiveness of this method heavily rely on a good row paving of the coefficient matrix. This motivates us to introduce a novel block selection strategy to the RBK method, called volume sampling, in which the probability of selection is proportional to the volume spanned by the rows of the selected submatrix. To further enhance the practical performance, we develop and analyze a momentum variant of the method. Convergence results are established and demonstrate the notable improvements in convergence factor of the RBK method brought by the volume sampling and the momentum acceleration. Furthermore, to efficiently implement the RBK method with volume sampling, we propose an efficient algorithm that enables volume sampling from a sparse matrix with sampling complexity that is only logarithmic in dimension. Numerical experiments confirm our theoretical results.

Paper Structure

This paper contains 27 sections, 13 theorems, 103 equations, 3 figures, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Related work
  4. Organization
  5. Basic definitions and Preliminaries
  6. Basic definitions
  7. Some useful lemmas
  8. Randomized block Kaczmarz with volume sampling
  9. The linear convergence of the expected norm of the error
  10. The linear convergence of the norm of the expected error
  11. Acceleration by heavy ball momentum
  12. Convergence of iterates
  13. Accelerated linear rate for expected iterates
  14. Implementation of volume sampling
  15. Preprocessing step
  16. ...and 12 more sections

Key Result

Lemma 2.2

Suppose $A$ is an $m \times n$ matrix with the singular value decomposition given by $A = U \Sigma V^\top$. Define $\lambda := (\sigma^2_1(A), \ldots,\sigma^2_{\operatorname{rank}(A)}(A),0,\ldots,0 )^\top \in\mathbb{R}^m$, which represents the eigenvalues of $AA^\top$. Assume $1 \leq s \leq \operato where and $\lambda_{-i} \in \mathbb{R}^{m-1}$ denotes the vector $\lambda$ with the $i$-th element

Figures (3)

  • Figure 1: Performance of mRBKVS with different momentum parameters $\beta$ and Type I coefficient matrices. We set $\sigma_{2}=10$ and $\delta=0.1$. The title of each plot indicates the values of $m$, $n, r$, and $\sigma_{1}/\sigma_{2}$.
  • Figure 2: Performance of mRBKVS with different momentum parameters $\beta$ and Type II coefficient matrices. The title of each plot indicates the values of $m$, $n, r$, and $\kappa$.
  • Figure 3: The evolution of RSE with respect to the number of iterations and CPU time. The CPU times of the preprocessing step for the line graph and the cycle graph are $0.0660$ and $1.0923$, respectively. The title of each plot indicates the type of graph and the values of $n$.

Theorems & Definitions (25)

  • Definition 2.1: Volume sampling
  • Lemma 2.2
  • Lemma 2.3
  • Lemma 2.4
  • Lemma 2.5
  • proof
  • Theorem 3.1
  • proof
  • Theorem 3.2
  • proof
  • ...and 15 more