Accelerated stochastic approximation with state-dependent noise
Sasila Ilandarideva, Anatoli Juditsky, Guanghui Lan, Tianjiao Li
TL;DR
This work develops accelerated stochastic approximation methods for convex optimization with state-dependent gradient noise, a setting motivated by generalized linear regression and sparse recovery. It presents two non-Euclidean algorithms, SAGD and SGE, and proves that SGE, in particular, achieves optimal iteration and sample complexities under broader noise assumptions than SAGD, including heavy-tailed or discontinuous gradient observations. A multi-stage restart framework for SGE is proposed to handle problems with quadratic growth, and a sparse-recovery variant (SGE-SR) is developed that combines hard-thresholding with accelerated updates to attain favorable dependence on sparsity. Theoretical results are complemented by numerical experiments demonstrating the efficacy of the proposed methods in high-dimensional, structured settings. The work advances the understanding of stochastic optimization with state-dependent noise and offers practical algorithms with strong convergence guarantees for statistical estimation and sparse recovery problems.
Abstract
We consider a class of stochastic smooth convex optimization problems under rather general assumptions on the noise in the stochastic gradient observation. As opposed to the classical problem setting in which the variance of noise is assumed to be uniformly bounded, herein we assume that the variance of stochastic gradients is related to the "sub-optimality" of the approximate solutions delivered by the algorithm. Such problems naturally arise in a variety of applications, in particular, in the well-known generalized linear regression problem in statistics. However, to the best of our knowledge, none of the existing stochastic approximation algorithms for solving this class of problems attain optimality in terms of the dependence on accuracy, problem parameters, and mini-batch size. We discuss two non-Euclidean accelerated stochastic approximation routines--stochastic accelerated gradient descent (SAGD) and stochastic gradient extrapolation (SGE)--which carry a particular duality relationship. We show that both SAGD and SGE, under appropriate conditions, achieve the optimal convergence rate, attaining the optimal iteration and sample complexities simultaneously. However, corresponding assumptions for the SGE algorithm are more general; they allow, for instance, for efficient application of the SGE to statistical estimation problems under heavy tail noises and discontinuous score functions. We also discuss the application of the SGE to problems satisfying quadratic growth conditions, and show how it can be used to recover sparse solutions. Finally, we report on some simulation experiments to illustrate numerical performance of our proposed algorithms in high-dimensional settings.