Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Level Constrained First Order Methods for Function Constrained Optimization

Digvijay Boob, Qi Deng, Guanghui Lan

Abstract

We present a new feasible proximal gradient method for constrained optimization where both the objective and constraint functions are given by the summation of a smooth, possibly nonconvex function and a convex simple function. The algorithm converts the original problem into a sequence of convex subproblems. Formulating those subproblems requires the evaluation of at most one gradient value of the original objective and constraint functions. Either exact or approximate subproblem solutions can be computed efficiently in many cases. An important feature of the algorithm is the constraint level parameter. By carefully increasing this level for each subproblem, we provide a simple solution to overcome the challenge of bounding the Lagrangian multipliers and show that the algorithm follows a strictly feasible solution path till convergence to the stationary point. We develop a simple, proximal gradient descent type analysis, showing that the complexity bound of this new algorithm is comparable to gradient descent for the unconstrained setting, which is new in the literature. Exploiting this new design and analysis technique, we extend our algorithms to some more challenging constrained optimization problems where 1) the objective is a stochastic or finite-sum function, and 2) structured nonsmooth functions replace smooth components of both objective and constraint functions. Complexity results for these problems also seem to be new in the literature. Finally, our method can also be applied to convex function-constrained problems where we show complexities similar to the proximal gradient method.

Level Constrained First Order Methods for Function Constrained Optimization

Abstract

We present a new feasible proximal gradient method for constrained optimization where both the objective and constraint functions are given by the summation of a smooth, possibly nonconvex function and a convex simple function. The algorithm converts the original problem into a sequence of convex subproblems. Formulating those subproblems requires the evaluation of at most one gradient value of the original objective and constraint functions. Either exact or approximate subproblem solutions can be computed efficiently in many cases. An important feature of the algorithm is the constraint level parameter. By carefully increasing this level for each subproblem, we provide a simple solution to overcome the challenge of bounding the Lagrangian multipliers and show that the algorithm follows a strictly feasible solution path till convergence to the stationary point. We develop a simple, proximal gradient descent type analysis, showing that the complexity bound of this new algorithm is comparable to gradient descent for the unconstrained setting, which is new in the literature. Exploiting this new design and analysis technique, we extend our algorithms to some more challenging constrained optimization problems where 1) the objective is a stochastic or finite-sum function, and 2) structured nonsmooth functions replace smooth components of both objective and constraint functions. Complexity results for these problems also seem to be new in the literature. Finally, our method can also be applied to convex function-constrained problems where we show complexities similar to the proximal gradient method.
Paper Structure (28 sections, 34 theorems, 202 equations, 3 figures, 4 tables, 4 algorithms)

This paper contains 28 sections, 34 theorems, 202 equations, 3 figures, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Comparison with MBA method
  3. Comparison with generalized SQP
  4. Outline
  5. Notations and assumptions
  6. Notations.
  7. A proximal gradient method
  8. Dependence of $B$ on the constraint qualification.
  9. Convergence rate analysis of LCPG method.
  10. Stochastic optimization
  11. Level constrained stochastic proximal gradient
  12. Level constrained stochastic variance reduced gradient descent
  13. Smooth optimization of nonsmooth constrained problems
  14. Inexact LCPG
  15. Convergence analysis under an inexactness criterion
  16. ...and 13 more sections

Key Result

Proposition 1

Let $x$ be a local optimal solution of Problem prob:main. If it satisfies MFCQ eq:mfcq, then there is a vector $\lambda\in\mathbb{R}_{+}^{m}$ such that the KKT condition def:kkt-cond holds.

Figures (3)

  • Figure 1: The nonconvex constraint $\psi_1(x) \le \eta_1$ where $\eta_1 =3$. The dotted blue curves are the subproblem constraint for two different points. Since the MFCQ is violated at $(5,0)$, the subproblem reduces to a single feasible point at the limit point $(5,0)$.
  • Figure 2: The nonconvex constraint $\psi_1(x)\le \eta_1$ where $\eta_1 = 2.5$. The dotted blue curves are subproblem constraint for two different points. Since the MFCQ is satisfied, the limiting subproblem constraint at $(3,0)$ is still a full dimensional set with nonempty interior.
  • Figure 3: Comparison of LCPG, LCSPG and LCSVRG. The first row reports the results on covtype (left: $\sigma=0.4$; right: $\sigma=0.6$). The second row reports the results on real-sim (left: $\sigma=0.1$; right: $\sigma=0.2$).

Theorems & Definitions (81)

  • Definition 1: KKT condition
  • Definition 2: MFCQ
  • Proposition 1: Necessary condition
  • Definition 3
  • Definition 4
  • Lemma 1: Three-point inequality
  • Proposition 2
  • proof
  • Theorem 1
  • proof
  • ...and 71 more