Gradient Descent for Convex and Smooth Noisy Optimization
Feifei Hu, Mathieu Gerber
TL;DR
The paper addresses noisy convex optimization where the objective F(θ)=E[f(θ,Z)] is strictly convex but not globally L-smooth. It introduces gradient descent with backtracking line search (GD-BLS) as a robust optimizer under weak smoothness and finite-moment assumptions, and shows that a single-stage sample-average approach yields a rate of O_P(B^{-1/4}) when using n(B)~B^{1/2}. To accelerate learning, it proposes a retrospective multi-stage refinement (J steps) that reallocates remaining budget to progressively finer approximations of F, achieving rates of O_P(B^{- rac{1}{2}(1- heta)}) with θ=δ^J, and a generalized rate O_P(B^{- rac{α}{1+α}(1-ig( rac{2α}{1+3α}ig)^J)}) for α∈(0,1]. The results hold without tuning parameters to the specific F or f, and are illustrated by Poisson-regression-inspired examples where SG can fail but GD-BLS succeeds, highlighting practical robustness. Overall, the work provides a near-optimal, budget-aware framework for noisy optimization beyond the standard L-smooth regime, with meaningful implications for large-scale statistical learning where smoothness and variance conditions may be violated.
Abstract
We study the use of gradient descent with backtracking line search (GD-BLS) to solve the noisy optimization problem $θ_\star:=\mathrm{argmin}_{θ\in\mathbb{R}^d} \mathbb{E}[f(θ,Z)]$, imposing that the function $F(θ):=\mathbb{E}[f(θ,Z)]$ is strictly convex but not necessarily $L$-smooth. Assuming that $\mathbb{E}[\|\nabla_θf(θ_\star,Z)\|^2]<\infty$, we first prove that sample average approximation based on GD-BLS allows to estimate $θ_\star$ with an error of size $\mathcal{O}_{\mathbb{P}}(B^{-0.25})$, where $B$ is the available computational budget. We then show that we can improve upon this rate by stopping the optimization process earlier when the gradient of the objective function is sufficiently close to zero, and use the residual computational budget to optimize, again with GD-BLS, a finer approximation of $F$. By iteratively applying this strategy $J$ times, we establish that we can estimate $θ_\star$ with an error of size $\mathcal{O}_{\mathbb{P}}(B^{-\frac{1}{2}(1-δ^{J})})$, where $δ\in(1/2,1)$ is a user-specified parameter. More generally, we show that if $\mathbb{E}[\|\nabla_θf(θ_\star,Z)\|^{1+α}]<\infty$ for some known $α\in (0,1]$ then this approach, which can be seen as a retrospective approximation algorithm with a fixed computational budget, allows to learn $θ_\star$ with an error of size $\mathcal{O}_{\mathbb{P}}(B^{-\fracα{1+α}(1-δ^{J})})$, where $δ\in(2α/(1+3α),1)$ is a tuning parameter. Beyond knowing $α$, achieving the aforementioned convergence rates do not require to tune the algorithms parameters according to the specific functions $F$ and $f$ at hand, and we exhibit a simple noisy optimization problem for which stochastic gradient is not guaranteed to converge while the algorithms discussed in this work are.