Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Power of Generalized Smoothness in Stochastic Convex Optimization: First- and Zero-Order Algorithms

Aleksandr Lobanov, Alexander Gasnikov

TL;DR

The paper addresses stochastic convex optimization under generalized $(L_0,L_1)$-smoothness, developing first-order methods with clipping (ClipSGD) and normalization (NSGD) and extending the analysis to zero-order algorithms (ZO-ClipSGD, ZO-NSGD). By deriving convergence-summing terms that capture linear-rate behavior, it establishes iteration complexities such as $N = \tilde{O}(L_1 R \log(1/\\varepsilon) + L_1 c R^2/\\varepsilon)$ for ClipSGD and $N = \tilde{O}(L_1 R \log(1/\\varepsilon))$ for NSGD in the $L_0=0$ regime, with NSGD requiring a large batch $B$ in general. The generalized smoothness framework is shown to extend to zero-order methods, yielding analogous linear-rate summands at the cost of additional oracle calls and noise tolerance, supported by logistic-regression experiments that confirm practical linear convergence. Overall, the work demonstrates that generalized smoothness can enable linear convergence in stochastic convex optimization and opens new directions for biased-gradient and zero-order approaches.

Abstract

This paper is devoted to the study of stochastic optimization problems under the generalized smoothness assumption. By considering the unbiased gradient oracle in Stochastic Gradient Descent, we provide strategies to achieve in bounds the summands describing linear rate. In particular, in the case $L_0 = 0$, we obtain in the convex setup the iteration complexity: $N = \mathcal{O}\left(L_1R \log\frac{1}{\varepsilon} + \frac{L_1 c R^2}{\varepsilon}\right)$ for Clipped Stochastic Gradient Descent and $N = \mathcal{O}\left(L_1R \log\frac{1}{\varepsilon}\right)$ for Normalized Stochastic Gradient Descent. Furthermore, we generalize the convergence results to the case with a biased gradient oracle, and show that the power of $(L_0,L_1)$-smoothness extends to zero-order algorithms. Finally, we demonstrate the possibility of linear convergence in the convex setup through numerical experimentation, which has aroused some interest in the machine learning community.

Power of Generalized Smoothness in Stochastic Convex Optimization: First- and Zero-Order Algorithms

TL;DR

The paper addresses stochastic convex optimization under generalized -smoothness, developing first-order methods with clipping (ClipSGD) and normalization (NSGD) and extending the analysis to zero-order algorithms (ZO-ClipSGD, ZO-NSGD). By deriving convergence-summing terms that capture linear-rate behavior, it establishes iteration complexities such as for ClipSGD and for NSGD in the regime, with NSGD requiring a large batch in general. The generalized smoothness framework is shown to extend to zero-order methods, yielding analogous linear-rate summands at the cost of additional oracle calls and noise tolerance, supported by logistic-regression experiments that confirm practical linear convergence. Overall, the work demonstrates that generalized smoothness can enable linear convergence in stochastic convex optimization and opens new directions for biased-gradient and zero-order approaches.

Abstract

This paper is devoted to the study of stochastic optimization problems under the generalized smoothness assumption. By considering the unbiased gradient oracle in Stochastic Gradient Descent, we provide strategies to achieve in bounds the summands describing linear rate. In particular, in the case , we obtain in the convex setup the iteration complexity: for Clipped Stochastic Gradient Descent and for Normalized Stochastic Gradient Descent. Furthermore, we generalize the convergence results to the case with a biased gradient oracle, and show that the power of -smoothness extends to zero-order algorithms. Finally, we demonstrate the possibility of linear convergence in the convex setup through numerical experimentation, which has aroused some interest in the machine learning community.

Paper Structure

This paper contains 60 sections, 6 theorems, 206 equations, 3 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. Main Contributions
  3. Formal Setting and Assumptions
  4. Notations.
  5. Assumptions on objective function.
  6. Assumptions on gradient oracle.
  7. Paper Organization
  8. Related Works
  9. Algorithms under $(L_0,L_1)$-smoothness.
  10. Zero-order algorithms.
  11. Clipped Stochastic Gradient Descent
  12. Normalized Stochastic Gradient Descent
  13. Zero-Order Algorithms
  14. ZO-ClipSGD Method
  15. ZO-NSGD Method
  16. ...and 45 more sections

Key Result

Theorem 3.1

Let function $f$ satisfy Assumption ass:L0_L1_smooth ($(L_0,L_1)$-smoothness) and unbiased gradient oracle eq:gradient_oracle satisfy Assumption ass:bounded_var (bounded variance), then Algorithm algo:clipSGD with constant step size ${\eta_k = \eta \leq \left[4 (L_0 + L_1 c)\right]^{-1}}$ and arbitr where $0 \leq K < N$ is number of iterations for which $\|\nabla f(x^k)\| \leq \frac{c}{2}$ is sati

Figures (3)

  • Figure 1: Changing regimes demonstration
  • Figure 2: Linear convergence demonstration.
  • Figure 3: Comparison of the considered algorithms.

Theorems & Definitions (8)

  • Remark 1.3: Clarification regarding $L_0 = 0$
  • Theorem 3.1
  • Theorem 4.1
  • Lemma 5.1
  • Theorem 5.2
  • Lemma 5.3
  • Theorem 5.4
  • proof