Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Randomized Extended Bregman-Kaczmarz Method for Combined Optimization Problems

Zeyu Dong, Aqin Xiao, Guojian Yin, Junfeng Yin

TL;DR

The paper addresses combined optimization problems that blend data fidelity with regularization by introducing the adaptive randomized averaging block extended Bregman-Kaczmarz (aRABEBK) method. By employing residual-driven adaptive relaxation parameters, the method achieves aggressive yet stable updates within a block-averaging Bregman-Kaczmarz framework, and a rigorous convergence theory establishes expected linear rates with explicit constants. The approach outperforms existing Kaczmarz-type schemes (REBK, REABK, cRABEBK) in both synthetic sparse/min-norm LS tasks and MNIST image recovery, demonstrating faster convergence and greater robustness to problem structure. Overall, aRABEBK offers a flexible, parameter-free (in practice) acceleration strategy for large-scale inverse problems that leverage sparsity and regularization.

Abstract

Combined optimization problems that couple data-fidelity and regularization terms arise naturally in a wide range of inverse problems. In this paper, we study an adaptive randomized averaging block extended Bregman-Kaczmarz (aRABEBK) method for solving such problems. The proposed method incorporates iteration-wise relaxation parameters that are automatically adjusted using residual information, allowing for more aggressive step sizes without additional manual tuning. We establish a convergence theory for the proposed framework and derive expected linear convergence rate guarantees. Numerical experiments on both synthetic and real data sets for sparse and minimum-norm least-squares problems demonstrate that our aRABEBK method achieves faster convergence and improved robustness compared with state-of-the-art extended Kaczmarz and Bregman-Kaczmarz-type algorithms, including its nonadaptive counterpart.

Adaptive Randomized Extended Bregman-Kaczmarz Method for Combined Optimization Problems

TL;DR

The paper addresses combined optimization problems that blend data fidelity with regularization by introducing the adaptive randomized averaging block extended Bregman-Kaczmarz (aRABEBK) method. By employing residual-driven adaptive relaxation parameters, the method achieves aggressive yet stable updates within a block-averaging Bregman-Kaczmarz framework, and a rigorous convergence theory establishes expected linear rates with explicit constants. The approach outperforms existing Kaczmarz-type schemes (REBK, REABK, cRABEBK) in both synthetic sparse/min-norm LS tasks and MNIST image recovery, demonstrating faster convergence and greater robustness to problem structure. Overall, aRABEBK offers a flexible, parameter-free (in practice) acceleration strategy for large-scale inverse problems that leverage sparsity and regularization.

Abstract

Combined optimization problems that couple data-fidelity and regularization terms arise naturally in a wide range of inverse problems. In this paper, we study an adaptive randomized averaging block extended Bregman-Kaczmarz (aRABEBK) method for solving such problems. The proposed method incorporates iteration-wise relaxation parameters that are automatically adjusted using residual information, allowing for more aggressive step sizes without additional manual tuning. We establish a convergence theory for the proposed framework and derive expected linear convergence rate guarantees. Numerical experiments on both synthetic and real data sets for sparse and minimum-norm least-squares problems demonstrate that our aRABEBK method achieves faster convergence and improved robustness compared with state-of-the-art extended Kaczmarz and Bregman-Kaczmarz-type algorithms, including its nonadaptive counterpart.
Paper Structure (15 sections, 6 theorems, 107 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 15 sections, 6 theorems, 107 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and related work
  3. Main contributions
  4. Organization
  5. Preliminaries
  6. The Proposed Methods
  7. Review of the RABEBK Method and Its Constant-Relaxation Variant
  8. Adaptive RABEBK method
  9. Convergence Analysis
  10. Numerical experiments
  11. Experiments on synthetic data
  12. Gaussian matrix
  13. Experiments on structured random matrix
  14. Experiment on image recovery
  15. Conclusion

Key Result

Lemma 2.1

If $f:\mathbb{R}^n \rightarrow \mathbb{R}$ is $\alpha$-strongly convex, then its conjugate function $f^*$ is differentiable with a $1/\alpha$-Lipschitz-continuous gradient, i.e. which implies the estimate

Figures (7)

  • Figure 1: Gaussian sparse least-squares test under the same setting as Section \ref{['sec:numerical experiments']}: $A\in\mathbb{R}^{500\times 1000}$ with i.i.d. Gaussian entries; the ground-truth $\hat{x}$ has $s=\lceil 0.01n\rceil$ nonzeros and $\lambda=5$.
  • Figure 2: Curves of the relative error versus the number of iterations for the sparse least-squares case with Gaussian matrices. Top: overdetermined matrices; Bottom: underdetermined matrices.
  • Figure 3: The curves of relative error versus the number of iterations for overdetermined Gaussian matrices (top) and underdetermined Gaussian matrices (bottom) in the minimum-norm least-squares case.
  • Figure 4: The curves of relative error versus the number of iterations for overdetermined constructed matrices (top) and underdetermined constructed matrices (bottom) in the sparse least-squares setting.
  • Figure 5: The curves of relative error versus the number of iterations for overdetermined constructed matrices (top) and underdetermined constructed matrices (bottom) in the minimum-norm least-squares setting.
  • ...and 2 more figures

Theorems & Definitions (16)

  • Lemma 2.1: rockafellar1998variational
  • Definition 2.1: Bregman distance bregman1967relaxation
  • Lemma 2.2: schopfer2019linear
  • Proposition 3.1: Exact line search in block form
  • proof
  • Definition 4.1
  • Definition 4.2
  • Definition 4.3
  • Theorem 4.1: Schpfer2022ExtendedRK, Theorem 3.9
  • Remark 4.1
  • ...and 6 more