Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Relaxed Randomized Averaging Block Extended Bregman-Kaczmarz Method for Combined Optimization Problems

Zeyu Dong, Aqin Xiao, Guojian Yin, Junfeng Yin

TL;DR

This work advances randomized Kaczmarz-type methods by introducing a relaxed randomized averaging block extended Bregman-Kaczmarz (rRABEBK) framework for solving combined optimization problems that enforce linear constraints while promoting sparsity. By integrating an averaging block scheme with two relaxation parameters, the method achieves linear convergence in expectation and empirically outperforms existing algorithms (REBK, RABEBK, and related block variants) in both iteration complexity and runtime, across sparse, Gaussian, structured, and image-recovery problems. The authors provide explicit convergence constants and guidelines for parameter selection, and demonstrate benefits in inconsistent systems as well as sparse least-squares settings. The approach promises practical impact for large-scale inverse problems where sparsity and data misfit must be balanced efficiently.

Abstract

Randomized Kaczmarz-type methods are widely used for their simplicity and efficiency in solving large-scale linear systems and optimization problems. However, their applicability is limited when dealing with inconsistent systems or incorporating structural information such as sparsity. In this work, we propose a \emph{relaxed randomized averaging block extended Bregman-Kaczmarz} (rRABEBK) method for solving a broad class of combined optimization problems. The proposed method integrates an averaging block strategy with two relaxation parameters to accelerate convergence and enhance numerical stability. We establish a rigorous convergence theory showing that rRABEBK achieves linear convergence in expectation, with explicit constants that quantify the effect of the relaxation mechanism, and a provably faster rate than the classical randomized extended Bregman-Kaczmarz method. Our method can be readily adapted to sparse least-squares problems and extended to both consistent and inconsistent systems without modification. Complementary numerical experiments corroborate the theoretical findings and demonstrate that rRABEBK significantly outperforms the existing Kaczmarz-type algorithms in terms of both iteration complexity and computational efficiency, highlighting both its practical and theoretical advantages.

A Relaxed Randomized Averaging Block Extended Bregman-Kaczmarz Method for Combined Optimization Problems

TL;DR

This work advances randomized Kaczmarz-type methods by introducing a relaxed randomized averaging block extended Bregman-Kaczmarz (rRABEBK) framework for solving combined optimization problems that enforce linear constraints while promoting sparsity. By integrating an averaging block scheme with two relaxation parameters, the method achieves linear convergence in expectation and empirically outperforms existing algorithms (REBK, RABEBK, and related block variants) in both iteration complexity and runtime, across sparse, Gaussian, structured, and image-recovery problems. The authors provide explicit convergence constants and guidelines for parameter selection, and demonstrate benefits in inconsistent systems as well as sparse least-squares settings. The approach promises practical impact for large-scale inverse problems where sparsity and data misfit must be balanced efficiently.

Abstract

Randomized Kaczmarz-type methods are widely used for their simplicity and efficiency in solving large-scale linear systems and optimization problems. However, their applicability is limited when dealing with inconsistent systems or incorporating structural information such as sparsity. In this work, we propose a \emph{relaxed randomized averaging block extended Bregman-Kaczmarz} (rRABEBK) method for solving a broad class of combined optimization problems. The proposed method integrates an averaging block strategy with two relaxation parameters to accelerate convergence and enhance numerical stability. We establish a rigorous convergence theory showing that rRABEBK achieves linear convergence in expectation, with explicit constants that quantify the effect of the relaxation mechanism, and a provably faster rate than the classical randomized extended Bregman-Kaczmarz method. Our method can be readily adapted to sparse least-squares problems and extended to both consistent and inconsistent systems without modification. Complementary numerical experiments corroborate the theoretical findings and demonstrate that rRABEBK significantly outperforms the existing Kaczmarz-type algorithms in terms of both iteration complexity and computational efficiency, highlighting both its practical and theoretical advantages.

Paper Structure

This paper contains 12 sections, 5 theorems, 87 equations, 4 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Kaczmarz Method and Some Extensions
  4. The Proposed Methods
  5. A Randomized Averaging Block Extended Bregman-Kaczmarz Method
  6. A Randomized Averaging Block Extended Bregman-Kaczmarz Method with Relaxation Parameters
  7. Numerical experiments
  8. Experiments on Bregman Updates and the Necessity of Relaxation
  9. Numerical Experiments with Gaussian Random Matrices
  10. Numerical Experiments on Structured Random Matrices
  11. Experiment on image recovery
  12. Conclusion

Key Result

Lemma 2.1

If $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is $\mu$-strongly convex then the conjugate function $f^*$ is differentiable with a $1/\mu$ Lipschitz-continuous gradient, i.e. for all $x^{*},y^{*} \in \mathbb{R}^n$. Consequently,

Figures (4)

  • Figure 1: Convergence histories of REBK, RABEK, RABEBK, and the relaxed rRABEBK variants for a Bernoulli coefficient matrix of size $500\times 1000$.
  • Figure 2: Relative error as a function of the iteration count for Gaussian test matrices. The top panel reports results for overdetermined systems, whereas the bottom panel presents those for underdetermined systems.
  • Figure 3: The curves of relative error versus the number of iterations for overdetermined constructed matrices (top) and underdetermined constructed matrices (bottom).
  • Figure 4: Original MNIST image (left) and reconstructed images obtained by REBK and rRABEBK variants after 10,000 iterations. The numbers below each reconstruction indicate the PSNR values (in dB).

Theorems & Definitions (13)

  • Lemma 2.1: rockafellar1998variational
  • Definition 2.2: Bregman distance bregman1967relaxation
  • Lemma 2.3: schopfer2019linear
  • Definition 4.1
  • Definition 4.2
  • Definition 4.3
  • Theorem 4.4: Schpfer2022ExtendedRK, Theorem 3.9
  • Theorem 4.5
  • Proof 1
  • Remark 4.6
  • ...and 3 more