Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Direct Spectral Acceleration of First-Order Methods for Saddle Point Problems with Bilinear Coupling

Meng Li, Paul Grigas

TL;DR

This work proposes direct spectral acceleration for first-order primal--dual methods for a class of bilinear-coupled saddle point problems, including affinely constrained smooth strongly convex optimization and extensions with proximable dual terms.

Abstract

We study convex-concave saddle point problems with bilinear coupling, covering linearly constrained convex optimization and more general nonsmooth or constrained models via a proximable term in the dual objective. In linearly convergent regimes, we characterize how spectral properties of the coupling matrix and objective conditioning jointly determine the attainable linear rates. We propose direct spectral acceleration for first-order primal--dual methods for a class of bilinear-coupled saddle point problems, including affinely constrained smooth strongly convex optimization and extensions with proximable dual terms. The resulting algorithms distinguish objective-dominated and coupling matrix-dominated regimes and attain optimal linear convergence without Chebyshev inner loops or double-loop designs. We further develop stochastic block-coordinate extensions in the affinely constrained case with separable objectives; we also establish optimal linear rates matching the block-coordinate lower bound. For both deterministic and stochastic methods, we provide matching worst-case lower bounds via explicit finite-dimensional hard instances.

Direct Spectral Acceleration of First-Order Methods for Saddle Point Problems with Bilinear Coupling

TL;DR

This work proposes direct spectral acceleration for first-order primal--dual methods for a class of bilinear-coupled saddle point problems, including affinely constrained smooth strongly convex optimization and extensions with proximable dual terms.

Abstract

We study convex-concave saddle point problems with bilinear coupling, covering linearly constrained convex optimization and more general nonsmooth or constrained models via a proximable term in the dual objective. In linearly convergent regimes, we characterize how spectral properties of the coupling matrix and objective conditioning jointly determine the attainable linear rates. We propose direct spectral acceleration for first-order primal--dual methods for a class of bilinear-coupled saddle point problems, including affinely constrained smooth strongly convex optimization and extensions with proximable dual terms. The resulting algorithms distinguish objective-dominated and coupling matrix-dominated regimes and attain optimal linear convergence without Chebyshev inner loops or double-loop designs. We further develop stochastic block-coordinate extensions in the affinely constrained case with separable objectives; we also establish optimal linear rates matching the block-coordinate lower bound. For both deterministic and stochastic methods, we provide matching worst-case lower bounds via explicit finite-dimensional hard instances.
Paper Structure (31 sections, 44 theorems, 217 equations, 7 figures, 5 tables, 5 algorithms)

This paper contains 31 sections, 44 theorems, 217 equations, 7 figures, 5 tables, 5 algorithms.

Table of Contents

  1. Introduction
  2. Directly Accelerated Primal--Dual Algorithms
  3. Accelerated Primal--Dual Algorithm (x-Side)
  4. Accelerated Primal--Dual Algorithm (y-Side)
  5. Stochastic Block-Coordinate Directly Accelerated Primal--Dual Algorithms
  6. Stochastic Block-Coordinate Accelerated Primal--Dual Algorithm (x-Side)
  7. Stochastic Block-Coordinate Accelerated Primal--Dual Algorithm (y-Side)
  8. Lower Bound Results
  9. Numerical Experiments
  10. Compressed-Sensing-Type Experiment
  11. Non-Smooth Estimator of the Linear Time-Invariant System
  12. Conclusions
  13. Notations
  14. Summary of Main Algorithms
  15. Proof of Proposition \ref{['prop:saddle_cond_2']}
  16. ...and 16 more sections

Key Result

Proposition 1

For problem eq:upper bound prob under Assumption assum:Assumption Linear New, there exists a unique saddle point $(x^*,y^*)$. Furthermore, $(x^*,y^*)$ is a saddle point if and only if

Figures (7)

  • Figure 1: Results for the compressed-sensing-type (CST) experiment. Left: $\frac{s_{\max}^2}{s_{\min}^2} = 10^5$, $\frac{L}{\mu}=10^4$; right: $\frac{s_{\max}^2}{s_{\min}^2} = 10^6$, $\frac{L}{\mu}=10^3$.
  • Figure 2: Results for the CST experiment with SBC-DAPD. Left: $\frac{s_{\max}^2}{s_{\min}^2} = 10^4$, $\frac{L}{\mu}=10^5$; right: $\frac{s_{\max}^2}{s_{\min}^2} = 10^6$, $\frac{L}{\mu}=10^3$.
  • Figure 3: Results for the NSE experiment \ref{['eq:nse reg short']}. Left: $\frac{L}{\mu}=10^9$, $p=40$, $T=10$, $\frac{s_{\max}^2}{s_{\min}^2} \approx 2.7 \times 10^5$ in the simulation; right: $\frac{L}{\mu}=10^4$, $p=100$, $T=20$, $\frac{s_{\max}^2}{s_{\min}^2} \approx 1.3 \times 10^9$ in the simulation.
  • Figure 4: Results for the Compressed-sensing-type experiment with longer iterations. Left: $\frac{s_{\max}^2}{s_{\min}^2} = 10^5$, $\frac{L}{\mu}=10^4$; right: $\frac{s_{\max}^2}{s_{\min}^2} = 10^6$, $\frac{L}{\mu}=10^3$.
  • Figure 5: Convergences of KKT metrics for the compressed-sensing-type experiment, Left: $\frac{s_{\max}^2}{s_{\min}^2} = 10^5$, $\frac{L}{\mu}=10^4$; right: $\frac{s_{\max}^2}{s_{\min}^2} = 10^6$, $\frac{L}{\mu}=10^3$.
  • ...and 2 more figures

Theorems & Definitions (89)

  • Proposition 1
  • Theorem 2.1
  • Theorem 2.2
  • Remark 1
  • Theorem 3.1
  • Remark 2
  • Remark 3
  • Remark 4
  • Theorem 3.2
  • Remark 5
  • ...and 79 more