The Güler-type acceleration for proximal gradient, linearized augmented Lagrangian and linearized alternating direction method of multipliers

TL;DR

Problem addressed: accelerating gradient-based methods for convex composite optimization. Approach: apply Güler-type acceleration to PGM, LALM, and L-ADMM, leveraging a negative extrapolation term to design extrapolations; this yields three algorithms GPGM, GLALM, and GLADMM in a unified framework. Key contributions: GPGM and GLALM achieve the same $O(1/k^2)$ rates as the best existing methods, while GLADMM attains $O(1/N)$ total with an improved $O(1/N^2)$ partial rate; all improvements are validated on $\ell_1$-regularized logistic regression, quadratic programming, and compressive sensing. Significance: the framework offers improved efficiency and a path to stochastic variants, with broad implications for statistics, machine learning, and data mining.

Abstract

In this paper, we introduce the Güler-type acceleration technique and utilize it to propose three acceleration algorithms: the Güler-type accelerated proximal gradient method (GPGM), the Güler-type accelerated linearized augmented Lagrangian method (GLALM) and the Güler-type accelerated linearized alternating direction method of multipliers (GLADMM). The key idea behind these algorithms is to fully leverage the information of negative term \bm{$-\|x^k-\hat{x}^{k-1}\|^2$} in order to design the extrapolation step. This concept of using negative terms to improve acceleration can be extended to other algorithms as well. Moreover, the proposed GLALM and GLADMM enable simultaneous acceleration of both primal and dual variables. Additionally, GPGM and GLALM achieve the same convergence rate of $O(\frac{1}{k^2})$ with some existing results. Although GLADMM achieves the same total convergence rate of $O(\frac{1}{N})$ as in existing results, the partial convergence rate is improved from $O(\frac{1}{N^{3/2}})$ to $O(\frac{1}{N^2})$. To validate the effectiveness of our algorithms, we conduct numerical experiments on various problem instances, including the $\ell_1$ regularized logistic regression, quadratic programming, and compressive sensing. The experimental results indicate that our algorithms outperform existing methods in terms of efficiency. This also demonstrates the potential of the stochastic algorithmic versions of these algorithms in application areas such as statistics, machine learning, and data mining. Finally, it is worth noting that this paper aims to introduce how Güler's acceleration technique can be applied to gradient-based algorithms and to provide a unified and concise framework for their construction.

Figures (5)

• Figure 1: The upper bounds of GPGM and Nesterov's second APGM
• Figure 2: The upper bounds of GLALM and ALALM
• Figure 3: $\ell_1-logistic$$m=300, n=3000, s=30$
• Figure 6: quadratic programming $m=80, n=1000$
• Figure 9: Comparisons of 300 iterations of L-ADMM, AL-ADMM and GLADMM

Theorems & Definitions (20)

• Lemma 2.1: bauschke2011convex
• Lemma 2.2
• Lemma 2.3: xu2017accelerated
• Lemma 2.4: xu2017accelerated
• Theorem 3.1
• proof
• Remark 3.1
• Remark 3.2
• Lemma 4.1
• proof