Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Accelerated Proximal Bundle Method with Momentum

Zhuoqing Zheng, Junshan Yin, Shaofu Yang, Xuyang Wu

Abstract

Proximal bundle methods (PBM) are a powerful class of algorithms for convex optimization. Compared to gradient descent, PBM constructs more accurate surrogate models that incorporate gradients and function values from multiple past iterations, which leads to faster and more robust convergence. However, for smooth convex problems, PBM only achieves an O(1/k) convergence rate, which is inferior to the optimal O(1/k^2) rate. To bridge this gap, we propose an accelerated proximal bundle method (APBM) that integrates Nesterov's momentum into PBM. We prove that under standard assumptions, APBM achieves the optimal O(1/k^2) convergence rate. Numerical experiments demonstrate the effectiveness of the proposed APBM.

An Accelerated Proximal Bundle Method with Momentum

Abstract

Proximal bundle methods (PBM) are a powerful class of algorithms for convex optimization. Compared to gradient descent, PBM constructs more accurate surrogate models that incorporate gradients and function values from multiple past iterations, which leads to faster and more robust convergence. However, for smooth convex problems, PBM only achieves an O(1/k) convergence rate, which is inferior to the optimal O(1/k^2) rate. To bridge this gap, we propose an accelerated proximal bundle method (APBM) that integrates Nesterov's momentum into PBM. We prove that under standard assumptions, APBM achieves the optimal O(1/k^2) convergence rate. Numerical experiments demonstrate the effectiveness of the proposed APBM.

Paper Structure

This paper contains 11 sections, 4 theorems, 41 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Algorithm
  3. Algorithm development
  4. Candidates of the model $\hat{f}^k$
  5. Solving the subproblem \ref{['eq:APBM-x']}
  6. Convergence Analysis
  7. Numerical Example
  8. Conclusion
  9. Proof of Lemma \ref{['lemma:mod_sat_asm']}
  10. Proof of Theorem \ref{['theo:sublinear']}
  11. Proof for Corollary \ref{['cor:APBM']}

Key Result

Lemma 1

Suppose that $f$ is convex and differentiable. Then, the four models eq:pol_model--eq:two-cut satisfy Assumption asm:pri_model. $\blacktriangleleft$$\blacktriangleleft$

Figures (3)

  • Figure 3: Surrogate functions in the Polyak model \ref{['eq:pol_model']}, cutting-plane model \ref{['eq:cpm']}, and the Polyak cutting-plane model \ref{['eq:pcpm']}
  • Figure 4: Convergence performance in solving the least squares problem \ref{['problem:LS']}
  • Figure 5: Convergence performance (with restart scheme) in solving the least squares problem \ref{['problem:LS']}

Theorems & Definitions (8)

  • Remark 1
  • Lemma 1
  • proof
  • Theorem 1
  • proof
  • Lemma 2
  • Corollary 1
  • proof