Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

Guanghui Lan, Tianjiao Li

TL;DR

This paper introduces auto-conditioned, line-search-free primal-dual and ADMM frameworks for bilinear saddle point and linearly constrained problems. The AC-PDHG and AC-ADMM algorithms adapt stepsizes to local estimates of the operator norms, avoiding the need to compute global norms, and achieve optimal iteration complexities. The authors further extend these methods with acceleration (AC-APDHG, AC-AADMM), delivering fast convergence in smooth settings under a single-oracle model, and provide guidance for practical implementation, including a line-search-free initial setup and a guess-and-check procedure. Overall, the work delivers adaptive, parameter-free, convergent methods with strong theoretical guarantees and practical applicability to a broad class of constrained optimization problems.

Abstract

Line search procedures are often employed in primal-dual methods for bilinear saddle point problems, especially when the norm of the linear operator is large or difficult to compute. In this paper, we demonstrate that line search is unnecessary by introducing a novel primal-dual method, the auto-conditioned primal-dual hybrid gradient (AC-PDHG) method, which achieves optimal complexity for solving bilinear saddle point problems. AC-PDHG is fully adaptive to the linear operator, using only past iterates to estimate its norm. We further tailor AC-PDHG to solve linearly constrained problems, providing convergence guarantees for both the optimality gap and constraint violation. Moreover, we explore an important class of linearly constrained problems where both the objective and constraints decompose into two parts. By incorporating the design principles of AC-PDHG into the preconditioned alternating direction method of multipliers (ADMM), we propose the auto-conditioned alternating direction method of multipliers (AC-ADMM), which guarantees convergence based solely on one part of the constraint matrix and fully adapts to it, eliminating the need for line search. Finally, we extend both AC-PDHG and AC-ADMM to solve bilinear problems with an additional smooth term. By integrating these methods with a novel acceleration scheme, we attain optimal iteration complexities under the single-oracle setting.

Auto-conditioned primal-dual hybrid gradient method and alternating direction method of multipliers

TL;DR

This paper introduces auto-conditioned, line-search-free primal-dual and ADMM frameworks for bilinear saddle point and linearly constrained problems. The AC-PDHG and AC-ADMM algorithms adapt stepsizes to local estimates of the operator norms, avoiding the need to compute global norms, and achieve optimal iteration complexities. The authors further extend these methods with acceleration (AC-APDHG, AC-AADMM), delivering fast convergence in smooth settings under a single-oracle model, and provide guidance for practical implementation, including a line-search-free initial setup and a guess-and-check procedure. Overall, the work delivers adaptive, parameter-free, convergent methods with strong theoretical guarantees and practical applicability to a broad class of constrained optimization problems.

Abstract

Line search procedures are often employed in primal-dual methods for bilinear saddle point problems, especially when the norm of the linear operator is large or difficult to compute. In this paper, we demonstrate that line search is unnecessary by introducing a novel primal-dual method, the auto-conditioned primal-dual hybrid gradient (AC-PDHG) method, which achieves optimal complexity for solving bilinear saddle point problems. AC-PDHG is fully adaptive to the linear operator, using only past iterates to estimate its norm. We further tailor AC-PDHG to solve linearly constrained problems, providing convergence guarantees for both the optimality gap and constraint violation. Moreover, we explore an important class of linearly constrained problems where both the objective and constraints decompose into two parts. By incorporating the design principles of AC-PDHG into the preconditioned alternating direction method of multipliers (ADMM), we propose the auto-conditioned alternating direction method of multipliers (AC-ADMM), which guarantees convergence based solely on one part of the constraint matrix and fully adapts to it, eliminating the need for line search. Finally, we extend both AC-PDHG and AC-ADMM to solve bilinear problems with an additional smooth term. By integrating these methods with a novel acceleration scheme, we attain optimal iteration complexities under the single-oracle setting.
Paper Structure (15 sections, 14 theorems, 179 equations, 6 algorithms)

This paper contains 15 sections, 14 theorems, 179 equations, 6 algorithms.

Table of Contents

  1. Introduction
  2. Smoothing scheme and primal-dual methods for \ref{['eq:bilinear_saddle_point']}
  3. Alternating direction method of multipliers (ADMM) for \ref{['linear_two_parts_0']}
  4. Adaptive methods for bilinear saddle point problems
  5. Contributions and organization
  6. Notation
  7. An adaptive primal-dual hybrid gradient (PDHG) method
  8. AC-PDHG for bilinear saddle point problems
  9. AC-PDHG for linearly constrained problems
  10. An adaptive alternating direction method of multipliers (ADMM)
  11. Adaptive PDHG and ADMM with acceleration
  12. Auto-conditioned accelerated PDHG for bilinear saddle point problems
  13. AC-APDHG for linearly constrained problems
  14. Auto-conditioned accelerated ADMM
  15. Concluding remarks

Key Result

Lemma 2.1

\newlabeldual_three-point_lemma Let $\{x_t\}, \{\bar{x}_t\}$ and $\{y_t\}$ be generated by Algorithm alg:ac_primal_dual, we have for $t \geq 1$, Moreover, if $\tau_1 = 0$, we have

Theorems & Definitions (26)

  • Lemma 2.1
  • proof
  • Proposition 2.2
  • proof
  • Theorem 2.3
  • proof
  • Corollary 2.4
  • proof
  • Proposition 2.5
  • proof
  • ...and 16 more