Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting the Geometrically Decaying Step Size: Linear Convergence for Smooth or Non-Smooth Functions

Jihun Kim

TL;DR

The paper introduces a positive inverse condition number, a weaker geometric criterion, to guarantee linear convergence of subgradient descent with geometrically decaying step sizes for both smooth and non-smooth locally Lipschitz functions, without requiring convexity or sharpness. It formalizes the condition using Clarke subdifferentials and a defined set S, provides practical sufficient conditions and broad nonconvex examples where the condition holds, and presents two algorithms: one requiring the distance to a minimizer and a second that replaces it with a known small constant, both achieving linear convergence with rates depending on problem-constants. This work expands the applicability of fast convergence guarantees to a wider class of nonconvex problems and lays groundwork for extensions to stochastic, robust, and distributed settings. Overall, the results show that geometry captured by ¯μ>0 suffices to drive linear convergence under minimally informative step-size schemes.

Abstract

We revisit the geometrically decaying step size given a positive inverse condition number, under which a locally Lipschitz function shows linear convergence. The positivity does not require the function to satisfy convexity, weak convexity, quasar convexity, or sharpness, but instead amounts to a property strictly weaker than the assumptions used in existing works (e.g., weak convexity + sharpness). We propose a clean and simple subgradient descent algorithm that requires minimal knowledge of problem constants, applicable to either smooth or non-smooth functions.

Revisiting the Geometrically Decaying Step Size: Linear Convergence for Smooth or Non-Smooth Functions

TL;DR

The paper introduces a positive inverse condition number, a weaker geometric criterion, to guarantee linear convergence of subgradient descent with geometrically decaying step sizes for both smooth and non-smooth locally Lipschitz functions, without requiring convexity or sharpness. It formalizes the condition using Clarke subdifferentials and a defined set S, provides practical sufficient conditions and broad nonconvex examples where the condition holds, and presents two algorithms: one requiring the distance to a minimizer and a second that replaces it with a known small constant, both achieving linear convergence with rates depending on problem-constants. This work expands the applicability of fast convergence guarantees to a wider class of nonconvex problems and lays groundwork for extensions to stochastic, robust, and distributed settings. Overall, the results show that geometry captured by ¯μ>0 suffices to drive linear convergence under minimally informative step-size schemes.

Abstract

We revisit the geometrically decaying step size given a positive inverse condition number, under which a locally Lipschitz function shows linear convergence. The positivity does not require the function to satisfy convexity, weak convexity, quasar convexity, or sharpness, but instead amounts to a property strictly weaker than the assumptions used in existing works (e.g., weak convexity + sharpness). We propose a clean and simple subgradient descent algorithm that requires minimal knowledge of problem constants, applicable to either smooth or non-smooth functions.

Paper Structure

This paper contains 5 sections, 5 theorems, 23 equations, 1 figure, 2 algorithms.

Table of Contents

  1. Introduction
  2. Assumption for Linear Convergence
  3. Geometrically Decaying Step Size Design
  4. Conclusion
  5. Proof for Theorem \ref{['theorem2']}

Key Result

lemma 1

Suppose that $\langle u, x-x^*\rangle > 0$ holds for all $x^*\in \mathcal{X}^*, x\in S\setminus \mathcal{X}^*$, and any $u \in \partial^\circ f(x)$. Then, a point $s\in S$ is a minimizer of $f$ if and only if $0\in \partial^\circ f(s)$.

Figures (1)

  • Figure 1: Examples of functions with positive inverse condition numbers but without (a) convexity, (b) weak convexity, (c) quasar convexity, or (d) sharpness

Theorems & Definitions (13)

  • remark 1
  • lemma 1
  • proof
  • remark 2
  • lemma 2
  • proof
  • lemma 3
  • proof
  • theorem 1
  • proof
  • ...and 3 more