Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic First-Order Methods with Non-smooth and Non-Euclidean Proximal Terms for Nonconvex High-Dimensional Stochastic Optimization

Yue Xie, Jiawen Bi, Hongcheng Liu

TL;DR

The paper tackles high-dimensional stochastic nonconvex optimization by introducing dimension-insensitive stochastic first-order methods (DISFOMs) that permit non-Euclidean and non-smooth proximal terms. It develops two closed-form proximal options and analyzes gradient estimators (minibatch and variance reduction), proving that DISFOMs achieve $O((\log d)/\epsilon^4)$ and $O((\log d)^{2/3}/\epsilon^{10/3})$ sample complexities respectively for obtaining an $\epsilon$-stationary point, with the latter representing a leading dependence on the dimension $d$. The theoretical results are complemented by numerical experiments showing dimension-stability and competitiveness against traditional proximal SGD/SVRG and SMD methods. Overall, the approach provides a scalable pathway for nonconvex stochastic optimization in settings where dimension and non-smooth proximal terms would otherwise hinder performance.

Abstract

When the nonconvex problem is complicated by stochasticity, the sample complexity of stochastic first-order methods may depend linearly on the problem dimension, which is undesirable for large-scale problems. In this work, we propose dimension-insensitive stochastic first-order methods (DISFOMs) to address nonconvex optimization with expected-valued objective function. Our algorithms allow for non-Euclidean and non-smooth distance functions as the proximal terms. Under mild assumptions, we show that DISFOM using minibatches to estimate the gradient enjoys sample complexity of $ \mathcal{O} ( (\log d) / ε^4 ) $ to obtain an $ε$-stationary point. Furthermore, we prove that DISFOM employing variance reduction can sharpen this bound to $\mathcal{O} ( (\log d)^{2/3}/ε^{10/3} )$, which perhaps leads to the best-known sample complexity result in terms of $d$. We provide two choices of the non-smooth distance functions, both of which allow for closed-form solutions to the proximal step. Numerical experiments are conducted to illustrate the dimension insensitive property of the proposed frameworks.

Stochastic First-Order Methods with Non-smooth and Non-Euclidean Proximal Terms for Nonconvex High-Dimensional Stochastic Optimization

TL;DR

The paper tackles high-dimensional stochastic nonconvex optimization by introducing dimension-insensitive stochastic first-order methods (DISFOMs) that permit non-Euclidean and non-smooth proximal terms. It develops two closed-form proximal options and analyzes gradient estimators (minibatch and variance reduction), proving that DISFOMs achieve and sample complexities respectively for obtaining an -stationary point, with the latter representing a leading dependence on the dimension . The theoretical results are complemented by numerical experiments showing dimension-stability and competitiveness against traditional proximal SGD/SVRG and SMD methods. Overall, the approach provides a scalable pathway for nonconvex stochastic optimization in settings where dimension and non-smooth proximal terms would otherwise hinder performance.

Abstract

When the nonconvex problem is complicated by stochasticity, the sample complexity of stochastic first-order methods may depend linearly on the problem dimension, which is undesirable for large-scale problems. In this work, we propose dimension-insensitive stochastic first-order methods (DISFOMs) to address nonconvex optimization with expected-valued objective function. Our algorithms allow for non-Euclidean and non-smooth distance functions as the proximal terms. Under mild assumptions, we show that DISFOM using minibatches to estimate the gradient enjoys sample complexity of to obtain an -stationary point. Furthermore, we prove that DISFOM employing variance reduction can sharpen this bound to , which perhaps leads to the best-known sample complexity result in terms of . We provide two choices of the non-smooth distance functions, both of which allow for closed-form solutions to the proximal step. Numerical experiments are conducted to illustrate the dimension insensitive property of the proposed frameworks.
Paper Structure (16 sections, 16 theorems, 105 equations, 2 figures, 3 algorithms)

This paper contains 16 sections, 16 theorems, 105 equations, 2 figures, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Notation.
  4. Discussion of residual function
  5. Dimension-insensitive stochastic first-order methods
  6. Choices of $\phi$
  7. Choices of $G^k$
  8. Convergence and complexity analysis
  9. Key lemmas
  10. Analysis for Case 1
  11. Analysis for Case 2
  12. Numerical Experiment
  13. Conclusion
  14. Proofs
  15. Algorithm
  16. ...and 1 more sections

Key Result

lemma thmcounterlemma

Let ${\mathbf x} \in X$, then

Figures (2)

  • Figure 1: averaged gap $(f-f^*) / \Delta$ vs $\log(d)$ and residual vs $\log(d)$ for mini-batch
  • Figure 2: averaged gap $(f-f^*) / \Delta$ vs $\log(d)$ and residual vs $\log(d)$ for variance reduction

Theorems & Definitions (33)

  • definition thmcounterdefinition
  • lemma thmcounterlemma
  • proof
  • lemma thmcounterlemma
  • proof
  • lemma thmcounterlemma
  • proof
  • lemma thmcounterlemma
  • remark thmcounterremark
  • lemma thmcounterlemma
  • ...and 23 more