Variance Reduction and Low Sample Complexity in Stochastic Optimization via Proximal Point Method
Jiaming Liang
TL;DR
The paper tackles high-confidence guarantees for stochastic convex composite optimization under bounded-variance noise by introducing the stochastic proximal point method (SPPM). It combines a variance-reducing proximal subproblem solver (PSS) with a probability booster (PB) to achieve high-probability convergence using a constant prox-stepsize λ, avoiding decaying step sizes. The main theoretical contributions are a high-probability convergence result and a low-sample-complexity bound that scales with log(1/p) for the confidence parameter, with an overall gradient-sample complexity of O(max{κ log(κ/ε), κσ^2/(μ ε)} log(1/p) log(1/ε)). The framework achieves variance reduction without mini-batching and provides a practical, adaptive approach to stochastic optimization that relies only on bounded-variance noise assumptions. These results have implications for reliable stochastic optimization in settings where sub-Gaussian noise cannot be guaranteed, offering a principled, proximal-based pathway to robust convergence.
Abstract
High-probability guarantees in stochastic optimization are often obtained only under strong noise assumptions such as sub-Gaussian tails. We show that such guarantees can also be achieved under the weaker assumption of bounded variance by developing a stochastic proximal point method. This method combines a proximal subproblem solver, which inherently reduces variance, with a probability booster that amplifies per-iteration reliability into high-confidence results. The analysis demonstrates convergence with low sample complexity, without restrictive noise assumptions or reliance on mini-batching.