Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles

Amir Ali Farzin, Yuen-Man Pun, Iman Shames

TL;DR

This work introduces a random Gaussian smoothing zeroth-order optimization framework for minimizing quasar-convex and strongly quasar-convex functions in both unconstrained and constrained settings. By replacing gradients with a Gaussian-smoothed surrogate $f_\mu$ and a corresponding unbiased estimator $g_\mu$, the authors establish convergence guarantees and neighbourhood control, with variance reduction enabling arbitrarily small residuals. They define proximal-quasar-convexity to extend these results to constrained problems and derive matching or near-matching iteration bounds to first-order methods. Empirically, the Gaussian-smoothing ZO approach demonstrates competitive or superior performance to gradient descent in learning linear dynamical systems and related tasks, offering stability advantages in challenging settings. Overall, the results broaden the applicability of zeroth-order methods to nonconvex but structured objectives and suggest practical benefits for black-box optimization in ML and control contexts.

Abstract

This study explores the performance of a random Gaussian smoothing zeroth-order (ZO) scheme for minimising quasar-convex (QC) and strongly quasar-convex (SQC) functions in both unconstrained and constrained settings. For the unconstrained problem, we establish the ZO algorithm's convergence to a global minimum along with its complexity when applied to both QC and SQC functions. For the constrained problem, we introduce the new notion of proximal-quasar-convexity and prove analogous results to the unconstrained case. Specifically, we show the complexity bounds and the convergence of the algorithm to a neighbourhood of a global minimum whose size can be controlled under a variance reduction scheme. Theoretical findings are illustrated through investigating the performance of the algorithm applied to a range of problems in machine learning and optimisation. Specifically, we observe scenarios where the ZO method outperforms gradient descent. We provide a possible explanation for this phenomenon.

Minimisation of Quasar-Convex Functions Using Random Zeroth-Order Oracles

TL;DR

This work introduces a random Gaussian smoothing zeroth-order optimization framework for minimizing quasar-convex and strongly quasar-convex functions in both unconstrained and constrained settings. By replacing gradients with a Gaussian-smoothed surrogate and a corresponding unbiased estimator , the authors establish convergence guarantees and neighbourhood control, with variance reduction enabling arbitrarily small residuals. They define proximal-quasar-convexity to extend these results to constrained problems and derive matching or near-matching iteration bounds to first-order methods. Empirically, the Gaussian-smoothing ZO approach demonstrates competitive or superior performance to gradient descent in learning linear dynamical systems and related tasks, offering stability advantages in challenging settings. Overall, the results broaden the applicability of zeroth-order methods to nonconvex but structured objectives and suggest practical benefits for black-box optimization in ML and control contexts.

Abstract

This study explores the performance of a random Gaussian smoothing zeroth-order (ZO) scheme for minimising quasar-convex (QC) and strongly quasar-convex (SQC) functions in both unconstrained and constrained settings. For the unconstrained problem, we establish the ZO algorithm's convergence to a global minimum along with its complexity when applied to both QC and SQC functions. For the constrained problem, we introduce the new notion of proximal-quasar-convexity and prove analogous results to the unconstrained case. Specifically, we show the complexity bounds and the convergence of the algorithm to a neighbourhood of a global minimum whose size can be controlled under a variance reduction scheme. Theoretical findings are illustrated through investigating the performance of the algorithm applied to a range of problems in machine learning and optimisation. Specifically, we observe scenarios where the ZO method outperforms gradient descent. We provide a possible explanation for this phenomenon.
Paper Structure (25 sections, 17 theorems, 87 equations, 9 figures, 1 table, 2 algorithms)

This paper contains 25 sections, 17 theorems, 87 equations, 9 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Outline
  4. Notation:
  5. Problems of Interest And Framework
  6. Main Results
  7. Unconstrained Problem
  8. Constrained Problem
  9. ZO Methods Average Landscape in the Neighbourhood
  10. Numerical Examples
  11. Learning Linear Dynamical System
  12. Support Vector Machine with smoothed hinge loss function
  13. Conclusions And Future Research Directions
  14. Auxiliary Lemmas and more background material
  15. Proofs of the theorems and corrollaries
  16. ...and 10 more sections

Key Result

Lemma 1

Let function $f:\mathbb{R}^n\rightarrow\mathbb{R}$ satisfy Assumption def:lip_grad and be a $\gamma$-quasar-convex function with respect to some minimiser $x^\ast\in\mathbb{R}^n$, then $f_\mu$ satisfies the below inequality,

Figures (9)

  • Figure 1: Average of the LDSI objective function value
  • Figure 2: Average of the objective function value with sigmoid link function
  • Figure 3: Average of the objective function value with Leaky ReLU link function
  • Figure 4: Average of the objective function value with ReLU link function
  • Figure 5: Average of the $\bar{f}_{T,\sigma}$ objective function value
  • ...and 4 more figures

Theorems & Definitions (42)

  • Definition 1: Quasar-convex function
  • Definition 2: Proximal $\gamma$-quasar-convex Functions
  • Lemma 1
  • Theorem 1
  • Corollary 1
  • Lemma 2
  • Theorem 2
  • Corollary 2
  • Corollary 3
  • Theorem 3
  • ...and 32 more