Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

Georgii Bychkov, Darina Dvinskikh, Anastasia Antsiferova, Alexander Gasnikov, Aleksandr Lobanov

TL;DR

A novel gradient-free algorithm to solve a convex stochastic optimization problem, such as those encountered in medicine, physics, and machine learning, where the objective function can only be computed through numerical simulation.

Abstract

We present a novel gradient-free algorithm to solve a convex stochastic optimization problem, such as those encountered in medicine, physics, and machine learning (e.g., adversarial multi-armed bandit problem), where the objective function can only be computed through numerical simulation, either as the result of a real experiment or as feedback given by the function evaluations from an adversary. Thus we suppose that only a black-box access to the function values of the objective is available, possibly corrupted by adversarial noise: deterministic or stochastic. The noisy setup can arise naturally from modeling randomness within a simulation or by computer discretization, or when exact values of function are forbidden due to privacy issues, or when solving non-convex problems as convex ones with an inexact function oracle. By exploiting higher-order smoothness, fulfilled, e.g., in logistic regression, we improve the performance of zero-order methods developed under the assumption of classical smoothness (or having a Lipschitz gradient). The proposed algorithm enjoys optimal oracle complexity and is designed under an overparameterization setup, i.e., when the number of model parameters is much larger than the size of the training dataset. Overparametrized models fit to the training data perfectly while also having good generalization and outperforming underparameterized models on unseen data. We provide convergence guarantees for the proposed algorithm under both types of noise. Moreover, we estimate the maximum permissible adversarial noise level that maintains the desired accuracy in the Euclidean setup, and then we extend our results to a non-Euclidean setup. Our theoretical results are verified on the logistic regression problem.

Accelerated zero-order SGD under high-order smoothness and overparameterized regime

TL;DR

A novel gradient-free algorithm to solve a convex stochastic optimization problem, such as those encountered in medicine, physics, and machine learning, where the objective function can only be computed through numerical simulation.

Abstract

We present a novel gradient-free algorithm to solve a convex stochastic optimization problem, such as those encountered in medicine, physics, and machine learning (e.g., adversarial multi-armed bandit problem), where the objective function can only be computed through numerical simulation, either as the result of a real experiment or as feedback given by the function evaluations from an adversary. Thus we suppose that only a black-box access to the function values of the objective is available, possibly corrupted by adversarial noise: deterministic or stochastic. The noisy setup can arise naturally from modeling randomness within a simulation or by computer discretization, or when exact values of function are forbidden due to privacy issues, or when solving non-convex problems as convex ones with an inexact function oracle. By exploiting higher-order smoothness, fulfilled, e.g., in logistic regression, we improve the performance of zero-order methods developed under the assumption of classical smoothness (or having a Lipschitz gradient). The proposed algorithm enjoys optimal oracle complexity and is designed under an overparameterization setup, i.e., when the number of model parameters is much larger than the size of the training dataset. Overparametrized models fit to the training data perfectly while also having good generalization and outperforming underparameterized models on unseen data. We provide convergence guarantees for the proposed algorithm under both types of noise. Moreover, we estimate the maximum permissible adversarial noise level that maintains the desired accuracy in the Euclidean setup, and then we extend our results to a non-Euclidean setup. Our theoretical results are verified on the logistic regression problem.

Paper Structure

This paper contains 16 sections, 2 theorems, 78 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Paper organization.
  3. Preliminaries
  4. Notation.
  5. Main results: the AZO-SGD-HS algorithm
  6. Deterministic adversarial noise
  7. Stochastic adversarial noise
  8. Experiments
  9. Synthetic data.
  10. Setup.
  11. Conclusion
  12. Appendix
  13. Proof of Theorem \ref{['teo:convergence_deterministic']}
  14. Proof of Remark \ref{['rem:convergence_deterministic']}
  15. Proof of Theorem \ref{['teo:convergence_stochastic']}
  16. ...and 1 more sections

Key Result

Theorem 1

Let $f(\cdot,\cdot)$ satisfy Assumption ass:High-smoothness with parameter $\beta$. Let smoothing parameter be $h \le \left({\varepsilon}/{(\kappa_{\beta}LR})\right)^{{1}/{(\beta - 1)}}$ then Assumption assum:overparam is satisfied with gradient approximation eq_batched_grad. Let the maximum admissi Let $x_N^{ag}$ be the output of the AZO-SGD-HS then in at most the following number of iterations

Figures (1)

  • Figure 1: Comparison of the proposed algorithm AZO-SGD-HS with AZO-SGD from lobanov2023accelerated

Theorems & Definitions (4)

  • Theorem 1
  • Remark 1
  • Theorem 2
  • Remark 2