Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Optimization on a Finer Scale: Bounded Local Subgradient Variation Perspective

Jelena Diakonikolas, Cristóbal Guzmán

Abstract

We initiate the study of nonsmooth optimization problems under bounded local subgradient variation, which postulates bounded difference between (sub)gradients in small local regions around points, in either average or maximum sense. The resulting class of objective functions encapsulates the classes of objective functions traditionally studied in optimization, which are defined based on either Lipschitz continuity of the objective or Hölder/Lipschitz continuity of its gradient. Further, the defined class contains functions that are neither Lipschitz continuous nor have a Hölder continuous gradient. When restricted to the traditional classes of optimization problems, the parameters defining the studied classes lead to more fine-grained complexity bounds, recovering traditional oracle complexity bounds in the worst case but generally leading to lower oracle complexity for functions that are not ``worst case.'' Some highlights of our results are that: (i) it is possible to obtain complexity results for both convex and nonconvex problems with the (local or global) Lipschitz constant being replaced by a constant of local subgradient variation and (ii) mean width of the subdifferential set around the optima plays a role in the complexity of nonsmooth optimization, particularly in parallel settings. A consequence of (ii) is that for any error parameter $ε> 0$, parallel oracle complexity of nonsmooth Lipschitz convex optimization is lower than its sequential oracle complexity by a factor $\tildeΩ\big(\frac{1}ε\big)$ whenever the objective function is piecewise linear with polynomially many pieces in the input size. This is particularly surprising as existing parallel complexity lower bounds are based on such classes of functions. The seeming contradiction is resolved by considering the region in which the algorithm is allowed to query the objective.

Optimization on a Finer Scale: Bounded Local Subgradient Variation Perspective

Abstract

We initiate the study of nonsmooth optimization problems under bounded local subgradient variation, which postulates bounded difference between (sub)gradients in small local regions around points, in either average or maximum sense. The resulting class of objective functions encapsulates the classes of objective functions traditionally studied in optimization, which are defined based on either Lipschitz continuity of the objective or Hölder/Lipschitz continuity of its gradient. Further, the defined class contains functions that are neither Lipschitz continuous nor have a Hölder continuous gradient. When restricted to the traditional classes of optimization problems, the parameters defining the studied classes lead to more fine-grained complexity bounds, recovering traditional oracle complexity bounds in the worst case but generally leading to lower oracle complexity for functions that are not ``worst case.'' Some highlights of our results are that: (i) it is possible to obtain complexity results for both convex and nonconvex problems with the (local or global) Lipschitz constant being replaced by a constant of local subgradient variation and (ii) mean width of the subdifferential set around the optima plays a role in the complexity of nonsmooth optimization, particularly in parallel settings. A consequence of (ii) is that for any error parameter , parallel oracle complexity of nonsmooth Lipschitz convex optimization is lower than its sequential oracle complexity by a factor whenever the objective function is piecewise linear with polynomially many pieces in the input size. This is particularly surprising as existing parallel complexity lower bounds are based on such classes of functions. The seeming contradiction is resolved by considering the region in which the algorithm is allowed to query the objective.
Paper Structure (28 sections, 17 theorems, 79 equations, 1 figure, 1 algorithm)

This paper contains 28 sections, 17 theorems, 79 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Motivation & Intuition
  3. Main Results
  4. Local subgradient variation as a measure of complexity
  5. Deterministic convex optimization under BVG$_{\hbox{\footnotesize max}}$
  6. Randomized and possibly parallel convex optimization under BVG$_{\hbox{\footnotesize max}}$ and BVG$_{\hbox{\footnotesize avg}}$
  7. Nonconvex optimization under BVG$_{\hbox{\footnotesize max}}$
  8. Related Work
  9. Smoothing approaches
  10. Local Lipschitzness and Relaxations
  11. Stronger oracles
  12. Optimization algorithms
  13. Preliminaries
  14. Bounded Local Variation of the Subgradient
  15. Bounded Maximum Local Variation
  16. ...and 13 more sections

Key Result

Theorem 2.1

If $f:\mathbb{R}^d\to\mathbb{R}$ is locally Lipschitz then it is differentiable almost everywhere.

Figures (1)

  • Figure 1: Two nonsmooth, piecewise-linear, functions with the same Lipschitz constant. If both functions are averaged over a radius $r$ shown on the right plot, the right function remains much closer to its averaged, smoothed, variant indicated by the dashed line. Further, on any local neighborhood of radius $r$, the right function exhibits much milder changes in the slope than the left function does on its (only) kink.

Theorems & Definitions (39)

  • Theorem 2.1: Rademacher Rademacher:1919
  • Theorem 2.2: Fundamental Theorem of Calculus (FTC)
  • Definition 3.1: BVG$_{\hbox{\footnotesize max}}$
  • Definition 3.2: Bounded Mean Oscillation
  • Definition 3.3: BVG$_{\hbox{\footnotesize avg}}$
  • Lemma 3.4
  • Lemma 3.5
  • proof
  • Lemma 3.6
  • proof
  • ...and 29 more