Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

New Primal-Dual Algorithm for Convex Problems

Shuning Liu, Zexian Liu

TL;DR

The paper addresses convex-concave saddle point problems with bilinear coupling by introducing a new primal-dual framework (NPDA) that uses two proximal terms based on current and previous iterates. It develops accelerated (ANPDA) and line-search (NPDAL) variants, proving global convergence and ergodic rates of $O(1/N)$, with $O(1/N^2)$ acceleration when the dual function is strongly convex. Theoretical results show that NPDA reduces to classical PDA and GRPDA under certain parameter settings, while NPDAL provides convergence guarantees without requiring knowledge of the spectral norm $L= orm{K}$. Numerical experiments on matrix games and LASSO demonstrate substantial performance gains over state-of-the-art primal-dual and ADMM-type methods. These methods offer robust, scalable tools for a broad class of convex-concave problems with bilinear coupling.

Abstract

Primal-dual algorithm (PDA) is a classic and popular scheme for convex-concave saddle point problems. It is universally acknowledged that the proximal terms in the subproblems about the primal and dual variables are crucial to the convergence theory and numerical performance of primal-dual algorithms. By taking advantage of the information from the current and previous iterative points, we exploit two new proximal terms for the subproblems about the primal and dual variables. Based on two new proximal terms, we present a new primal-dual algorithm for convex-concave saddle point problems with bilinear coupling terms and establish its global convergence and O(1/N ) ergodic convergence rate. When either the primal function or the dual function is strongly convex, we accelerate the above proposed algorithm and show that the corresponding algorithm can achieve O(1/N^2) convergence rate. Since the conditions for the stepsizes of the proposed algorithm are related directly to the spectral norm of the linear transform, which is difficult to obtain in some applications, we also introduce a linesearch strategy for the above proposed primal-dual algorithm and establish its global convergence and O(1/N ) ergodic convergence rate . Some numerical experiments are conducted on matrix game and LASSO problems by comparing with other state-of-the-art algorithms, which demonstrate the effectiveness of the proposed three primal-dual algorithms.

New Primal-Dual Algorithm for Convex Problems

TL;DR

The paper addresses convex-concave saddle point problems with bilinear coupling by introducing a new primal-dual framework (NPDA) that uses two proximal terms based on current and previous iterates. It develops accelerated (ANPDA) and line-search (NPDAL) variants, proving global convergence and ergodic rates of , with acceleration when the dual function is strongly convex. Theoretical results show that NPDA reduces to classical PDA and GRPDA under certain parameter settings, while NPDAL provides convergence guarantees without requiring knowledge of the spectral norm . Numerical experiments on matrix games and LASSO demonstrate substantial performance gains over state-of-the-art primal-dual and ADMM-type methods. These methods offer robust, scalable tools for a broad class of convex-concave problems with bilinear coupling.

Abstract

Primal-dual algorithm (PDA) is a classic and popular scheme for convex-concave saddle point problems. It is universally acknowledged that the proximal terms in the subproblems about the primal and dual variables are crucial to the convergence theory and numerical performance of primal-dual algorithms. By taking advantage of the information from the current and previous iterative points, we exploit two new proximal terms for the subproblems about the primal and dual variables. Based on two new proximal terms, we present a new primal-dual algorithm for convex-concave saddle point problems with bilinear coupling terms and establish its global convergence and O(1/N ) ergodic convergence rate. When either the primal function or the dual function is strongly convex, we accelerate the above proposed algorithm and show that the corresponding algorithm can achieve O(1/N^2) convergence rate. Since the conditions for the stepsizes of the proposed algorithm are related directly to the spectral norm of the linear transform, which is difficult to obtain in some applications, we also introduce a linesearch strategy for the above proposed primal-dual algorithm and establish its global convergence and O(1/N ) ergodic convergence rate . Some numerical experiments are conducted on matrix game and LASSO problems by comparing with other state-of-the-art algorithms, which demonstrate the effectiveness of the proposed three primal-dual algorithms.

Paper Structure

This paper contains 13 sections, 9 theorems, 85 equations, 8 figures, 3 algorithms.

Table of Contents

  1. Introduction.
  2. Related algorithms.
  3. Motivations and contributions.
  4. Organization.
  5. Preliminaries.
  6. The proposed algorithms.
  7. New primal-dual algorithm (NPDA) for \ref{['p']} when $g(x)$ and $f^*(y)$ are generally convex.
  8. Accelerated NPDA for \ref{['p']} when $f^*(y)$ is strongly convex.
  9. NPDA with linesearch.
  10. Numerical experiment.
  11. Matrix game.
  12. LASSO.
  13. Conclusion.

Key Result

Lemma 1

For any extended real-valued closed proper convex function $h: X\rightarrow(-\infty,+\infty]$, $\tau >0$ and $x\in X$, it holds that $z=\text{Prox}_{\tau h}(x)$ if and only if $\langle z-x,y-z \rangle \geq \tau ( h(z)-h(y) )$ for $\forall y\in X$, namely,

Figures (8)

  • Figure 1: Example 1
  • Figure 2: Example 2
  • Figure 3: Example 3
  • Figure 4: Example 4
  • Figure 6: Example 1
  • ...and 3 more figures

Theorems & Definitions (22)

  • Lemma 1: w8, Fact 2.1
  • Lemma 2: w15, Fact 2.1
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Theorem 1: global convergence
  • proof
  • Theorem 2: ergodic convergence
  • proof
  • ...and 12 more