Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A stochastic perturbed augmented Lagrangian method for smooth convex constrained minimization

Nitesh Kumar Singh, Ion Necoara

TL;DR

This work introduces SGDPA, a stochastic perturbed augmented Lagrangian method for smooth convex optimization with many functional constraints. By alternating a stochastic primal update on a randomly chosen constraint with a perturbed dual ascent, SGDPA achieves sublinear rates under mild assumptions and attains improved rates when dual iterates are bounded or provably bounded. Theoretical results establish convergence in both optimality and feasibility, with rates ranging from $O(k^{-1/2})$ in the bounded-dual convex setting to $O(k^{-1/4})$ when duals are bounded via perturbation, and analogous rates for strongly convex objectives. Numerical experiments on convex QCQPs and MPC problems indicate that SGDPA is competitive with, and often faster than, state-of-the-art methods such as LALM, PDSG, and FICO, especially in large-scale constraint settings.

Abstract

This paper considers smooth convex optimization problems with many functional constraints. To solve this general class of problems we propose a new stochastic perturbed augmented Lagrangian method, called SGDPA, where a perturbation is introduced in the augmented Lagrangian function by multiplying the dual variables with a subunitary parameter. Essentially, we linearize the objective and one randomly chosen functional constraint within the perturbed augmented Lagrangian at the current iterate and add a quadratic regularization that leads to a stochastic gradient descent update for the primal variables, followed by a perturbed random coordinate ascent step to update the dual variables. We provide a convergence analysis in both optimality and feasibility criteria for the iterates of SGDPA algorithm using basic assumptions on the problem. In particular, when the dual updates are assumed to be bounded, we prove sublinear rates of convergence for the iterates of algorithm SGDPA of order $\mathcal{O} (k^{-1/2})$ when the objective is convex and of order $\mathcal{O} (k^{-1})$ when the objective is strongly convex, where $k$ is the iteration counter. Under some additional assumptions, we prove that the dual iterates are bounded and in this case we obtain convergence rates of order $\mathcal{O} (k^{-1/4})$ and $\mathcal{O} (k^{-1/2})$ when the objective is convex and strongly convex, respectively. Preliminary numerical experiments on problems with many quadratic constraints demonstrate the viability and performance of our method when compared to some existing state-of-the-art optimization methods and software.

A stochastic perturbed augmented Lagrangian method for smooth convex constrained minimization

TL;DR

This work introduces SGDPA, a stochastic perturbed augmented Lagrangian method for smooth convex optimization with many functional constraints. By alternating a stochastic primal update on a randomly chosen constraint with a perturbed dual ascent, SGDPA achieves sublinear rates under mild assumptions and attains improved rates when dual iterates are bounded or provably bounded. Theoretical results establish convergence in both optimality and feasibility, with rates ranging from in the bounded-dual convex setting to when duals are bounded via perturbation, and analogous rates for strongly convex objectives. Numerical experiments on convex QCQPs and MPC problems indicate that SGDPA is competitive with, and often faster than, state-of-the-art methods such as LALM, PDSG, and FICO, especially in large-scale constraint settings.

Abstract

This paper considers smooth convex optimization problems with many functional constraints. To solve this general class of problems we propose a new stochastic perturbed augmented Lagrangian method, called SGDPA, where a perturbation is introduced in the augmented Lagrangian function by multiplying the dual variables with a subunitary parameter. Essentially, we linearize the objective and one randomly chosen functional constraint within the perturbed augmented Lagrangian at the current iterate and add a quadratic regularization that leads to a stochastic gradient descent update for the primal variables, followed by a perturbed random coordinate ascent step to update the dual variables. We provide a convergence analysis in both optimality and feasibility criteria for the iterates of SGDPA algorithm using basic assumptions on the problem. In particular, when the dual updates are assumed to be bounded, we prove sublinear rates of convergence for the iterates of algorithm SGDPA of order when the objective is convex and of order when the objective is strongly convex, where is the iteration counter. Under some additional assumptions, we prove that the dual iterates are bounded and in this case we obtain convergence rates of order and when the objective is convex and strongly convex, respectively. Preliminary numerical experiments on problems with many quadratic constraints demonstrate the viability and performance of our method when compared to some existing state-of-the-art optimization methods and software.

Paper Structure

This paper contains 16 sections, 18 theorems, 117 equations, 1 figure, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Notations and assumptions
  3. Perturbed augmented Lagrangian and its properties
  4. Stochastic gradient descent and perturbed ascent method
  5. Convergence analysis of SGDPA
  6. Convergence analysis when the dual iterates are assumed to be bounded
  7. Convergence rates when the objective is convex
  8. Convergence rates when the objective is strongly convex
  9. Convergence analysis when the dual iterates are proved to be bounded
  10. Convergence rates when the objective is convex
  11. Convergence rates when the objective is strongly convex
  12. Numerical experiments
  13. Selection of initial stepsize
  14. Solving synthetic convex QCQPs
  15. Model predictive control with ellipsoidal state constraints
  16. ...and 1 more sections

Key Result

Lemma 1

Let us assume that the bounds from eq:Bddness hold. Then, choosing $\rho > 0, \tau \in [0,1)$ and $x \in \mathcal{Y}$, we have the following relation true:

Figures (1)

  • Figure 1: Behaviour of SGDPA for two choices of $\tau$ ($0$ and $\epsilon=10^{-2}$) and comparison with LALM and PDSG in terms of optimality (left) and feasibility (right) for $n=100, m = 100$ along the number of epochs.

Theorems & Definitions (19)

  • Definition 1
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Lemma 6
  • Lemma 7
  • Theorem 1
  • Theorem 2
  • ...and 9 more