Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Zeroth-order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization

Spyridon Pougkakiotis, Dionysios S. Kalogerias

TL;DR

This work tackles zeroth-order optimization for stochastic problems with weakly convex, potentially nonsmooth components by building a smooth surrogate $f_{\mu}$ via Gaussian smoothing and applying a proximal stochastic gradient method to $\phi_{\mu}=f_{\mu}+r$. It proves a convergence rate of $\mathcal{O}(n^2\epsilon^{-4})$ iterations to an $\epsilon$-stationary point of the surrogate Moreau envelope $\phi_{\mu}^{1/\bar{\rho}}$, with an interpretable $\mathcal{O}(\sqrt{n}/T^{1/4})$ form in terms of total iterations. The paper also demonstrates practical viability through phase retrieval experiments and introduces a novel hyper-parameter tuning framework that optimizes ADMM penalties for PDE-constrained problems, achieving good out-of-sample performance. The results suggest that the proposed zeroth-order approach is competitive with existing methods and offers a scalable, first-order-information-free tool for challenging stochastic optimization tasks with broad applicability.

Abstract

In this paper we analyze a zeroth-order proximal stochastic gradient method suitable for the minimization of weakly convex stochastic optimization problems. We consider nonsmooth and nonlinear stochastic composite problems, for which (sub-)gradient information might be unavailable. The proposed algorithm utilizes the well-known Gaussian smoothing technique, which yields unbiased zeroth-order gradient estimators of a related partially smooth surrogate problem (in which one of the two nonsmooth terms in the original problem's objective is replaced by a smooth approximation). This allows us to employ a standard proximal stochastic gradient scheme for the approximate solution of the surrogate problem, which is determined by a single smoothing parameter, and without the utilization of first-order information. We provide state-of-the-art convergence rates for the proposed zeroth-order method using minimal assumptions. The proposed scheme is numerically compared against alternative zeroth-order methods as well as a stochastic sub-gradient scheme on a standard phase retrieval problem. Further, we showcase the usefulness and effectiveness of our method for the unique setting of automated hyper-parameter tuning. In particular, we focus on automatically tuning the parameters of optimization algorithms by minimizing a novel heuristic model. The proposed approach is tested on a proximal alternating direction method of multipliers for the solution of $\mathcal{L}_1/\mathcal{L}_2$-regularized PDE-constrained optimal control problems, with evident empirical success.

A Zeroth-order Proximal Stochastic Gradient Method for Weakly Convex Stochastic Optimization

TL;DR

This work tackles zeroth-order optimization for stochastic problems with weakly convex, potentially nonsmooth components by building a smooth surrogate via Gaussian smoothing and applying a proximal stochastic gradient method to . It proves a convergence rate of iterations to an -stationary point of the surrogate Moreau envelope , with an interpretable form in terms of total iterations. The paper also demonstrates practical viability through phase retrieval experiments and introduces a novel hyper-parameter tuning framework that optimizes ADMM penalties for PDE-constrained problems, achieving good out-of-sample performance. The results suggest that the proposed zeroth-order approach is competitive with existing methods and offers a scalable, first-order-information-free tool for challenging stochastic optimization tasks with broad applicability.

Abstract

In this paper we analyze a zeroth-order proximal stochastic gradient method suitable for the minimization of weakly convex stochastic optimization problems. We consider nonsmooth and nonlinear stochastic composite problems, for which (sub-)gradient information might be unavailable. The proposed algorithm utilizes the well-known Gaussian smoothing technique, which yields unbiased zeroth-order gradient estimators of a related partially smooth surrogate problem (in which one of the two nonsmooth terms in the original problem's objective is replaced by a smooth approximation). This allows us to employ a standard proximal stochastic gradient scheme for the approximate solution of the surrogate problem, which is determined by a single smoothing parameter, and without the utilization of first-order information. We provide state-of-the-art convergence rates for the proposed zeroth-order method using minimal assumptions. The proposed scheme is numerically compared against alternative zeroth-order methods as well as a stochastic sub-gradient scheme on a standard phase retrieval problem. Further, we showcase the usefulness and effectiveness of our method for the unique setting of automated hyper-parameter tuning. In particular, we focus on automatically tuning the parameters of optimization algorithms by minimizing a novel heuristic model. The proposed approach is tested on a proximal alternating direction method of multipliers for the solution of -regularized PDE-constrained optimal control problems, with evident empirical success.
Paper Structure (20 sections, 8 theorems, 49 equations, 4 figures, 6 algorithms)

This paper contains 20 sections, 8 theorems, 49 equations, 4 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Notation
  3. Structure of the article
  4. Preliminaries
  5. Stochastic weakly convex functions
  6. Gaussian smoothing
  7. Proximal point and the Moreau envelope
  8. A zeroth-order proximal stochastic gradient method
  9. Convergence analysis
  10. Numerical results
  11. Phase retrieval
  12. Hyper-parameter tuning for optimization methods
  13. Proximal ADMM for PDE-constrained optimization
  14. Automated tuning: problem formulation and numerical results
  15. Poisson optimal control
  16. ...and 5 more sections

Key Result

Proposition 1

Any $\rho$-weakly convex function $f \colon \mathbb{R}^n \mapsto \mathbb{R}$ is locally Lipschitz continuous and regular in the sense of Clarke, and thus directionally differentiable. Furthermore, it is bounded below, and there exists $z \in \mathbb{R}^n$ such that Moreover, the latter holds for any $z \in \partial f(x_1)$. Finally, the map $x \mapsto f(x) + \frac{\rho}{2}\|x\|_2^2$ is convex and

Figures (4)

  • Figure 1: Convergence profiles for Z-ProxSG, DSZ-ProxSG, Uni-ZproxSG, SPSA and ProxSSG: average objective function value (lines) and 95% confidence intervals (shaded regions) vs number of iterations. The upper row corresponds, from left to right, to $(d,m) = (10,30),\ (20,45)$. The middle row corresponds, from left to right, to $(d,m) = (40,60),\ (35,90)$. The lower row corresponds, from left to right, to $(d,m) = (30,120),\ (80,150)$.
  • Figure 2: Convergence profiles for Z-ProxSG, DSZ-ProxSG: average objective function value (lines) and 95% confidence intervals (shaded regions) vs number of iterations, for $(d,m) = (40,60)$. The upper row corresponds, from left to right, to $(\mu_1,\mu_2) = (10^{-x},10^{-y})$, $x = 4, 5, 6$, $y = 7$. The lower row corresponds, from left to right, to $(\mu_1,\mu_2) = (10^{-x},10^{-y})$, $x = 6, 7, 8$, $y = 9$. In each case we set $\mu = \mu_2$.
  • Figure 3: Convergence profiles for pADMM with varying penalty parameter $\sigma$: average residual reduction (lines) and 95% confidence intervals (shaded regions) vs number of pADMM iterations. The algorithm is run over 40 randomly selected (out-of-sample) Poisson optimal control problems.
  • Figure 4: Convergence profiles for pADMM with varying penalty parameter $\sigma$: average residual reduction (lines) and 95% confidence intervals (shaded regions) vs number of pADMM iterations. The algorithm is run over 40 randomly selected (out-of-sample) convection-diffusion optimal control problems.

Theorems & Definitions (22)

  • Definition 1
  • Proposition 1
  • proof
  • Proposition 2
  • proof
  • Remark 1
  • Lemma 2.1
  • proof
  • Lemma 3.1
  • proof
  • ...and 12 more