A convex combination based primal-dual algorithm with linesearch for general convex-concave saddle point problems

TL;DR

A novel primal-dual algorithm for solving structured convex-concave saddle point problems with a generic smooth nonbilinear coupling term and an accelerated algorithm achieving the faster O(1/N^2) ergodic convergence rate for the strongly convex case.

Abstract

Using convex combination and linesearch techniques, we introduce a novel primal-dual algorithm for solving structured convex-concave saddle point problems with a generic smooth nonbilinear coupling term. Our adaptive linesearch strategy works under specific local smoothness conditions, allowing for potentially larger stepsizes. For an important class of structured convex optimization problems, the proposed algorithm reduces to a fully adaptive proximal gradient algorithm without linesearch, thereby representing an advancement over the golden ratio algorithm delineated in [Y. Malitsky, Math. Program. 2020]. We establish global pointwise and ergodic sublinear convergence rate of the algorithm measured by the primal-dual gap function in the general case. When the coupling term is linear in the dual variable, we measure the convergence rate by function value residual and constraint violation of an equivalent constrained optimization problem. Furthermore, an accelerated algorithm achieving the faster O(1/N^2) ergodic convergence rate is presented for the strongly convex case, where N denotes the iteration number. Our numerical experiments on quadratically constrained quadratic programming and sparse logistic regression problems indicate the new algorithm is significantly faster than the comparison algorithms.

Figures (7)

• Figure 1: The relation between $\psi$ and $\varphi$ determined by $\xi=\psi-\frac{\psi^3\varphi}{2(1+\psi)}$ with different values of $\xi$.
• Figure 2: Performance comparison of PDAc-L with $\beta$ being constant or adaptively determined by \ref{['adaptive-beta']}.
• Figure 3: Performance comparison of PDAc-L with $\eta=0$ and $\eta=0.9$.
• Figure 4: Details of linesearch for the problem \ref{['QCQP']} with $n=500$ and $m=10$. (a) Comparison results of extra linesearch trial steps taken by PDB and PDAc-L. (b) Cumulative results for both algorithms.
• Figure 5: Comparison results in terms of function value residual (left) and feasibility violations (right) on 10 random QCQPs with $n=500$ and $m=10$. First row: convergence as the iterations proceeded. Second row: convergence as CPU time proceeded.

Theorems & Definitions (27)

• Remark 2.1
• Remark 3.1
• Lemma 3.1
• proof
• Lemma 3.2
• proof
• Lemma 3.3
• proof
• Remark 3.2
• Theorem 3.1: Global pointwise convergence