Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

Feng-Yi Liao, Lijun Ding, Yang Zheng

TL;DR

The paper addresses linear convergence for nonsmooth, nonconvex optimization under weaker regularity conditions by extending the equivalence among strong convexity, restricted secant inequality, subdifferential error bound, Polyak-Łojasiewicz inequality, and quadratic growth to the class of $\rho$-weakly convex functions. It develops a simple, modular framework based on slope and Ekeland's variational principle to relate these conditions and to prove linear convergence of the proximal point method (PPM) and its inexact variant (iPPM) in convex and weakly convex settings. The key contributions include a complete characterization and coefficient conversions among the regularity conditions, a clean linear-convergence proof for PPM that extends to weakly convex functions with appropriate initialization, and a rigorous treatment of inexact subproblems with provable linear convergence under standard inexactness controls. The results broaden the applicability of fast convergence guarantees to a wide range of nonsmooth, weakly convex problems encountered in machine learning and signal processing, with practical numerical demonstrations on SVM, Lasso, and Elastic-Net problems.

Abstract

Many practical optimization problems lack strong convexity. Fortunately, recent studies have revealed that first-order algorithms also enjoy linear convergences under various weaker regularity conditions. While the relationship among different conditions for convex and smooth functions is well-understood, it is not the case for the nonsmooth setting. In this paper, we go beyond convexity and smoothness, and clarify the connections among common regularity conditions in the class of weakly convex functions, including $\textit{strong convexity}$, $\textit{restricted secant inequality}$, $\textit{subdifferential error bound}$, $\textit{Polyak-Łojasiewicz inequality}$, and $\textit{quadratic growth}$. In addition, using these regularity conditions, we present a simple and modular proof for the linear convergence of the proximal point method (PPM) for convex and weakly convex optimization problems. The linear convergence also holds when the subproblems of PPM are solved inexactly with a proper control of inexactness.

Error bounds, PL condition, and quadratic growth for weakly convex functions, and linear convergences of proximal point methods

TL;DR

The paper addresses linear convergence for nonsmooth, nonconvex optimization under weaker regularity conditions by extending the equivalence among strong convexity, restricted secant inequality, subdifferential error bound, Polyak-Łojasiewicz inequality, and quadratic growth to the class of -weakly convex functions. It develops a simple, modular framework based on slope and Ekeland's variational principle to relate these conditions and to prove linear convergence of the proximal point method (PPM) and its inexact variant (iPPM) in convex and weakly convex settings. The key contributions include a complete characterization and coefficient conversions among the regularity conditions, a clean linear-convergence proof for PPM that extends to weakly convex functions with appropriate initialization, and a rigorous treatment of inexact subproblems with provable linear convergence under standard inexactness controls. The results broaden the applicability of fast convergence guarantees to a wide range of nonsmooth, weakly convex problems encountered in machine learning and signal processing, with practical numerical demonstrations on SVM, Lasso, and Elastic-Net problems.

Abstract

Many practical optimization problems lack strong convexity. Fortunately, recent studies have revealed that first-order algorithms also enjoy linear convergences under various weaker regularity conditions. While the relationship among different conditions for convex and smooth functions is well-understood, it is not the case for the nonsmooth setting. In this paper, we go beyond convexity and smoothness, and clarify the connections among common regularity conditions in the class of weakly convex functions, including , , , , and . In addition, using these regularity conditions, we present a simple and modular proof for the linear convergence of the proximal point method (PPM) for convex and weakly convex optimization problems. The linear convergence also holds when the subproblems of PPM are solved inexactly with a proper control of inexactness.
Paper Structure (25 sections, 14 theorems, 77 equations, 3 figures, 1 table)

This paper contains 25 sections, 14 theorems, 77 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Motivation and preliminaries
  3. Motivation: Linear convergence of gradient descent algorithms
  4. The notion of slope and Ekeland's variational principle
  5. Relationships between regularity conditions
  6. Regularity conditions and their relationship
  7. Examples
  8. Proof of \ref{['prop:QG-EB-PL']}
  9. Proximal point method for convex optimization
  10. Proximal point method
  11. (Sub)linear convergences of PPM
  12. Inexact proximal point method (iPPM) and its convergence
  13. iPPM and stopping criteria
  14. (Sub)linear convergence of iPPM
  15. Applications
  16. ...and 10 more sections

Key Result

Theorem 2.1

Consider the optimization problem $\min_{x \in \mathbb{R}^n} f(x),$ where $f$ is an $L$-smooth function. Suppose its solution set $S$ is nonempty. If RSI eq:RSI-smooth holds with $\mu_{\mathrm{r}} > 0$ and PL inequality eq:PL-smooth holds with $\mu_{\mathrm{p}} > 0$, then the GD algorithm eq:GD_iter with $\omega_1 = \sqrt{1-\mu_{\mathrm{r}}^2/L^2} \in (0,1)$ and $\omega_2 = (L^3 - 2\mu_{\mathrm{r}

Figures (3)

  • Figure 1: A nonconvex function with $f^\star = 0$ that satisfies the equivalency $\ref{['eq:RSI']}\equiv \ref{['eq:EB']} \equiv \ref{['eq:PL']} \equiv \ref{['eq:QG']}$. Left: the function $f$ (blue line) is $2$-weakly convex (confirmed by the green line), and also satisfies \ref{['eq:QG']} with $\mu_{\mathrm{q}} = 5$ (confirmed by the red line); Middle: $f$ satisfies \ref{['eq:EB']} as $\mathrm{dist}(0, \hat{\partial} f(x))$ (yellow curve) is lower bounded by $\mathrm{dist}(x,S)$ (purple curve). Right: $f$ satisfies \ref{['eq:PL']} as $\mathrm{dist}^2(0,\hat{\partial} f(x))$ (red curve) is lower bounded by $f(x)$ (blue curve).
  • Figure 2: Linear convergences of the distance to the solution set (left) and cost value gaps (right) for a $2$-weakly convex function in \ref{['example:weakly-convex-function']}.
  • Figure 3: Linear convergences of cost value gaps for linear SVM (left), lasso (middle), and elastic-net (right).

Theorems & Definitions (24)

  • Theorem 2.1: Linear convergence of GD
  • Lemma 2.1
  • Lemma 2.2: Lemma 2.5 drusvyatskiy2015curves
  • Theorem 2.2: Ekeland's variational principle ekeland1974variational
  • Theorem 3.1
  • Remark 1
  • Example 1
  • Theorem 4.1: Sublinear convergence guler1991convergence
  • Theorem 4.2: Linear convergence
  • proof
  • ...and 14 more