Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Smoothed Primal-Dual Algorithms for Nonconvex Optimization with Linear Inequality Constraints

Ruichuan Huang, Jiawei Zhang, Ahmet Alacaoglu

TL;DR

This work introduces a stochastic, smoothed primal–dual framework for nonconvex optimization with linear inequality constraints, using a Moreau-envelope-based analysis to enable a single-sample SGD at each iteration. The proposed stochastic smoothed linearized ALM achieves an optimal O(ε^{-4}) oracle complexity for obtaining an ε-near stationary point, with a postprocessing step to guarantee ε-stationarity, and extends to stochastic linear constraints with safeguarding. By incorporating variance reduction (STORM), the authors achieve O(ε^{-3}) complexity under a stronger variance-reduction assumption, matching known lower bounds in the unconstrained case. The framework is demonstrated across distributed optimization, discrete relaxations, and fairness-constrained classification, illustrating its broad applicability to structured stochastic nonconvex problems.

Abstract

We propose smoothed primal-dual algorithms for solving stochastic and smooth nonconvex optimization problems with linear inequality constraints. Our algorithms are single-loop and only require a single stochastic gradient based on one sample at each iteration. A distinguishing feature of our algorithm is that it is based on an inexact gradient descent framework for the Moreau envelope, where the gradient of the Moreau envelope is estimated using one step of a stochastic primal-dual augmented Lagrangian method. To handle inequality constraints and stochasticity, we combine the recently established global error bounds in constrained optimization with a Moreau envelope-based analysis of stochastic proximal algorithms. For obtaining $\varepsilon$-stationary points, we establish the optimal $O(\varepsilon^{-4})$ sample complexity guarantee for our algorithms and provide extensions to stochastic linear constraints. We also show how to improve this complexity to $O(\varepsilon^{-3})$ by using variance reduction and the expected smoothness assumption. Unlike existing methods, the iterations of our algorithms are free of subproblems, large batch sizes or increasing penalty parameters and use dual variable updates to ensure feasibility.

Stochastic Smoothed Primal-Dual Algorithms for Nonconvex Optimization with Linear Inequality Constraints

TL;DR

This work introduces a stochastic, smoothed primal–dual framework for nonconvex optimization with linear inequality constraints, using a Moreau-envelope-based analysis to enable a single-sample SGD at each iteration. The proposed stochastic smoothed linearized ALM achieves an optimal O(ε^{-4}) oracle complexity for obtaining an ε-near stationary point, with a postprocessing step to guarantee ε-stationarity, and extends to stochastic linear constraints with safeguarding. By incorporating variance reduction (STORM), the authors achieve O(ε^{-3}) complexity under a stronger variance-reduction assumption, matching known lower bounds in the unconstrained case. The framework is demonstrated across distributed optimization, discrete relaxations, and fairness-constrained classification, illustrating its broad applicability to structured stochastic nonconvex problems.

Abstract

We propose smoothed primal-dual algorithms for solving stochastic and smooth nonconvex optimization problems with linear inequality constraints. Our algorithms are single-loop and only require a single stochastic gradient based on one sample at each iteration. A distinguishing feature of our algorithm is that it is based on an inexact gradient descent framework for the Moreau envelope, where the gradient of the Moreau envelope is estimated using one step of a stochastic primal-dual augmented Lagrangian method. To handle inequality constraints and stochasticity, we combine the recently established global error bounds in constrained optimization with a Moreau envelope-based analysis of stochastic proximal algorithms. For obtaining -stationary points, we establish the optimal sample complexity guarantee for our algorithms and provide extensions to stochastic linear constraints. We also show how to improve this complexity to by using variance reduction and the expected smoothness assumption. Unlike existing methods, the iterations of our algorithms are free of subproblems, large batch sizes or increasing penalty parameters and use dual variable updates to ensure feasibility.

Paper Structure

This paper contains 34 sections, 29 theorems, 217 equations, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Lagrangian, penalty and augmented Lagrangian functions
  3. Primal vs primal-dual algorithms.
  4. Contributions
  5. Preliminaries
  6. Stationary points.
  7. Assumptions
  8. Algorithm
  9. Interpretation 1: Linearized proximal ALM.
  10. Interpretation 2: Inexact GD on the Moreau envelope.
  11. Convergence Analysis
  12. Main Theorem
  13. Analysis Tools
  14. Proof Sketch
  15. One iteration inequality on the potential function
  16. ...and 19 more sections

Key Result

Theorem 3.1

Let asmp: 1 hold and run Algorithm alg: sgd with the parameters given in eq: alg_params (see also eq: param_defs_appendix). We have that $\mathbb{E}\| \nabla \Psi(\mathbf{z}_{t^*})\|\leq\varepsilon$ where $t^*$ is selected uniformly at random from $\{1, \dots, T\}$ with $T=\Theta(\varepsilon^{-4})$.

Theorems & Definitions (54)

  • Theorem 3.1
  • Corollary 3.2
  • Lemma 3.3
  • Lemma 3.4
  • Lemma 3.5
  • Lemma 3.6
  • Remark 4.1
  • Theorem 4.3
  • Remark 5.1
  • Lemma 5.3
  • ...and 44 more