Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An accelerated randomized Bregman-Kaczmarz method for strongly convex linearly constraint optimization

Lionel Tondji, Dirk A. Lorenz, Ion Necoara

TL;DR

This work tackles solving $\min f(x)$ subject to $\mathbf{A}x=b$ with a strongly convex, potentially non-smooth $f$, using only one row of $\mathbf{A}$ per iteration. By exploiting a dual-coordinate-descent view and transferring acceleration to the primal space, the authors propose the Accelerated Randomized Bregman-Kaczmarz (ARBK) method, achieving $\mathcal{O}(1/k^2)$ convergence in expectation. They provide a rigorous convergence analysis linking dual and primal iterates and demonstrate accelerated performance over standard Bregman-Kaczmarz and randomized Kaczmarz variants in synthetic and real-data experiments, particularly for sparse solutions. The method offers a scalable, efficient approach for large-scale linear constraint problems with strong empirical benefits and practical Python implementations.

Abstract

In this paper, we propose a randomized accelerated method for the minimization of a strongly convex function under linear constraints. The method is of Kaczmarz-type, i.e. it only uses a single linear equation in each iteration. To obtain acceleration we build on the fact that the Kaczmarz method is dual to a coordinate descent method. We use a recently proposed acceleration method for the randomized coordinate descent and transfer it to the primal space. This method inherits many of the attractive features of the accelerated coordinate descent method, including its worst-case convergence rates. A theoretical analysis of the convergence of the proposed method is given. Numerical experiments show that the proposed method is more efficient and faster than the existing methods for solving the same problem.

An accelerated randomized Bregman-Kaczmarz method for strongly convex linearly constraint optimization

TL;DR

This work tackles solving subject to with a strongly convex, potentially non-smooth , using only one row of per iteration. By exploiting a dual-coordinate-descent view and transferring acceleration to the primal space, the authors propose the Accelerated Randomized Bregman-Kaczmarz (ARBK) method, achieving convergence in expectation. They provide a rigorous convergence analysis linking dual and primal iterates and demonstrate accelerated performance over standard Bregman-Kaczmarz and randomized Kaczmarz variants in synthetic and real-data experiments, particularly for sparse solutions. The method offers a scalable, efficient approach for large-scale linear constraint problems with strong empirical benefits and practical Python implementations.

Abstract

In this paper, we propose a randomized accelerated method for the minimization of a strongly convex function under linear constraints. The method is of Kaczmarz-type, i.e. it only uses a single linear equation in each iteration. To obtain acceleration we build on the fact that the Kaczmarz method is dual to a coordinate descent method. We use a recently proposed acceleration method for the randomized coordinate descent and transfer it to the primal space. This method inherits many of the attractive features of the accelerated coordinate descent method, including its worst-case convergence rates. A theoretical analysis of the convergence of the proposed method is given. Numerical experiments show that the proposed method is more efficient and faster than the existing methods for solving the same problem.

Paper Structure

This paper contains 9 sections, 5 theorems, 32 equations, 5 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. Contribution
  4. Outline.
  5. Notation and basic notions
  6. Coordinate descent and Bregman-Kaczmarz method
  7. Convergence results for accelerated Randomized Bregman-Kaczmarz method
  8. Experiments
  9. CONCLUSIONS

Key Result

Lemma IV.1

fercoq2016restarting The sequence $(\theta_k)$ defined by $\theta_0 \leq 1$ and $\theta_{k+1} = \frac{\sqrt{\theta_k^4 + 4\theta_k^2} - \theta_k^2}{2}$ satisfies

Figures (5)

  • Figure 1: A comparison of randomized sparse Kaczmarz (blue), Nesterov acceleration scheme (green) and ARBK method (red), $m = 700, n = 700,$ sparsity $s=182$, $\lambda=30$, $\kappa(A) =1150$.
  • Figure 2: A comparison of randomized sparse Kaczmarz (blue), Nesterov acceleration scheme (green) and ARBK method (red), $m = 900, n = 200,$ sparsity $s=65$, $\lambda=30$, $\kappa(A) =2.70$.
  • Figure 3: A comparison of randomized sparse Kaczmarz (blue), Nesterov acceleration scheme (green) and ARBK method (red), $m = 500, n = 784,$ sparsity $s=7$, $\lambda=60$, $\kappa(A) =8.98$.
  • Figure 4: A comparison of randomized sparse Kaczmarz (blue), Nesterov acceleration scheme (green) and ARBK method (red), $m = 300, n = 900,$ sparsity $s=231$, $\lambda=15$, $\kappa(A) =2990$.
  • Figure 5: A comparison of randomized sparse Kaczmarz(RSK) (blue), Nesterov acceleration scheme(NRSK) (green) and ARBK method (red). On top from left to the right, the original picture, the reconstruction of RSK. On the bottom from left to right the reconstruction of ARBK and the reconstruction of NRSK.$m = 500, n = 784,$ sparsity $s=135$, $\lambda=30$, $\kappa(A) =8.98$.

Theorems & Definitions (11)

  • Definition II.1
  • Example II.1
  • Remark III.1
  • Lemma IV.1
  • Lemma IV.2
  • Lemma IV.3
  • proof
  • Theorem IV.4
  • proof
  • Theorem IV.5
  • ...and 1 more