Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

Nachuan Xiao, Xiaoyin Hu, Kim-Chuan Toh

TL;DR

This paper investigates the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function and proves that, for any sequence of sufficiently small stepsizes and approximation parameters, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion.

Abstract

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.

Stochastic Subgradient Methods with Guaranteed Global Stability in Nonsmooth Nonconvex Optimization

TL;DR

This paper investigates the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function and proves that, for any sequence of sufficiently small stepsizes and approximation parameters, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion.

Abstract

In this paper, we focus on providing convergence guarantees for stochastic subgradient methods in minimizing nonsmooth nonconvex functions. We first investigate the global stability of a general framework for stochastic subgradient methods, where the corresponding differential inclusion admits a coercive Lyapunov function. We prove that, for any sequence of sufficiently small stepsizes and approximation parameters, coupled with sufficiently controlled noises, the iterates are uniformly bounded and asymptotically stabilize around the stable set of its corresponding differential inclusion. Moreover, we develop an improved analysis to apply our proposed framework to establish the global stability of a wide range of stochastic subgradient methods, where the corresponding Lyapunov functions are possibly non-coercive. These theoretical results illustrate the promising potential of our proposed framework for establishing the global stability of various stochastic subgradient methods.
Paper Structure (39 sections, 32 theorems, 167 equations, 2 figures, 1 table)

This paper contains 39 sections, 32 theorems, 167 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. A general framework for stochastic subgradient methods
  3. Contributions
  4. Organization
  5. Preliminaries
  6. Notations
  7. Nonsmooth analysis
  8. Differential inclusion
  9. Global Stability for General Framework
  10. Basic assumptions and main results
  11. Subgradient methods using reshuffling
  12. Subgradient methods using with-replacement sampling
  13. Convergence Properties of SGD-type Methods
  14. Basic properties
  15. Global stability
  16. ...and 24 more sections

Key Result

Lemma 2.4

Suppose both $\mathcal{D}_{g}$ and $\mathcal{D}_h$ are locally bounded set-valued graph-closed mappings. Then the set-valued mapping $\mathcal{D}_g\circ \mathcal{D}_h$ is a locally bounded set-valued graph-closed mapping.

Figures (2)

  • Figure 1: Illustrations of the proof technique in Theorem \ref{['Theo_abstract_stability']}. Here the dash line refers to the interpolated process of $\{x_k: i\leq k\leq \Lambda(\lambda(i)+T)\}$, and the arrowed curve refers to a trajectory $\bar{x}^{\star}$ of the differential inclusion \ref{['Eq_stable_DI']}.
  • Figure 2: Test results on the performance of all the compared methods for training ResNet50 on CIFAR-10 and CIFAR-100 datasets.

Theorems & Definitions (77)

  • Definition 2.1
  • Definition 2.2
  • Definition 2.3
  • Lemma 2.4
  • Definition 2.5
  • Definition 2.6
  • Definition 2.7
  • Lemma 2.8: Theorem 1 and Corollary 1 in bolte2021conservative
  • Definition 2.9
  • Proposition 2.10
  • ...and 67 more