Adaptive Variance Reduction for Stochastic Optimization under Weaker Assumptions

TL;DR

A novel adaptive STORM method is introduced that achieves an optimal convergence rate of $\mathcal{O}(T^{-1/3})$ for non-convex functions with the authors' newly designed learning rate strategy.

Abstract

This paper explores adaptive variance reduction methods for stochastic optimization based on the STORM technique. Existing adaptive extensions of STORM rely on strong assumptions like bounded gradients and bounded function values, or suffer an additional $\mathcal{O}(\log T)$ term in the convergence rate. To address these limitations, we introduce a novel adaptive STORM method that achieves an optimal convergence rate of $\mathcal{O}(T^{-1/3})$ for non-convex functions with our newly designed learning rate strategy. Compared with existing approaches, our method requires weaker assumptions and attains the optimal convergence rate without the additional $\mathcal{O}(\log T)$ term. We also extend the proposed technique to stochastic compositional optimization, obtaining the same optimal rate of $\mathcal{O}(T^{-1/3})$. Furthermore, we investigate the non-convex finite-sum problem and develop another innovative adaptive variance reduction method that achieves an optimal convergence rate of $\mathcal{O}(n^{1/4} T^{-1/2} )$, where $n$ represents the number of component functions. Numerical experiments across various tasks validate the effectiveness of our method.

Figures (3)

• Figure 1: Results for CIFAR-10 dataset.
• Figure 2: Results for CIFAR-100 dataset.
• Figure 3: Results for WikiText-2 dataset.

Theorems & Definitions (23)

• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Lemma 1
• Proof 1
• Lemma 2
• Proof 2
• Lemma 3