Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stochastic Optimization with Constraints: A Non-asymptotic Instance-Dependent Analysis

Koulik Khamaru

TL;DR

This work addresses constrained stochastic convex optimization by developing a variance-reduced proximal gradient (VRPG) algorithm and proving a non-asymptotic, instance-dependent bound. The core idea links algorithm performance to a small perturbation of the original problem, quantified by a benchmark $\delta^2(N)$ that captures noise and constraint geometry, and shows that as $N$ grows, the VRPG bound aligns with the Hájek–Le Cam local minimax limit up to universal constants and a $\log N$ factor. The results provide practical, problem-structure-aware guarantees beyond worst-case analyses and suggest near-optimality in the asymptotic regime, while highlighting avenues to further remove logarithmic terms. Overall, the paper advances understanding of how constraint geometry and stochastic noise shape the finite-sample performance of variance-reduced methods in constrained settings.

Abstract

We consider the problem of stochastic convex optimization under convex constraints. We analyze the behavior of a natural variance reduced proximal gradient (VRPG) algorithm for this problem. Our main result is a non-asymptotic guarantee for VRPG algorithm. Contrary to minimax worst case guarantees, our result is instance-dependent in nature. This means that our guarantee captures the complexity of the loss function, the variability of the noise, and the geometry of the constraint set. We show that the non-asymptotic performance of the VRPG algorithm is governed by the scaled distance (scaled by $\sqrt{N}$) between the solutions of the given problem and that of a certain small perturbation of the given problem -- both solved under the given convex constraints; here, $N$ denotes the number of samples. Leveraging a well-established connection between local minimax lower bounds and solutions to perturbed problems, we show that as $N \rightarrow \infty$, the VRPG algorithm achieves the renowned local minimax lower bound by Hàjek and Le Cam up to universal constants and a logarithmic factor of the sample size.

Stochastic Optimization with Constraints: A Non-asymptotic Instance-Dependent Analysis

TL;DR

This work addresses constrained stochastic convex optimization by developing a variance-reduced proximal gradient (VRPG) algorithm and proving a non-asymptotic, instance-dependent bound. The core idea links algorithm performance to a small perturbation of the original problem, quantified by a benchmark that captures noise and constraint geometry, and shows that as grows, the VRPG bound aligns with the Hájek–Le Cam local minimax limit up to universal constants and a factor. The results provide practical, problem-structure-aware guarantees beyond worst-case analyses and suggest near-optimality in the asymptotic regime, while highlighting avenues to further remove logarithmic terms. Overall, the paper advances understanding of how constraint geometry and stochastic noise shape the finite-sample performance of variance-reduced methods in constrained settings.

Abstract

We consider the problem of stochastic convex optimization under convex constraints. We analyze the behavior of a natural variance reduced proximal gradient (VRPG) algorithm for this problem. Our main result is a non-asymptotic guarantee for VRPG algorithm. Contrary to minimax worst case guarantees, our result is instance-dependent in nature. This means that our guarantee captures the complexity of the loss function, the variability of the noise, and the geometry of the constraint set. We show that the non-asymptotic performance of the VRPG algorithm is governed by the scaled distance (scaled by ) between the solutions of the given problem and that of a certain small perturbation of the given problem -- both solved under the given convex constraints; here, denotes the number of samples. Leveraging a well-established connection between local minimax lower bounds and solutions to perturbed problems, we show that as , the VRPG algorithm achieves the renowned local minimax lower bound by Hàjek and Le Cam up to universal constants and a logarithmic factor of the sample size.
Paper Structure (27 sections, 4 theorems, 59 equations, 1 algorithm)

This paper contains 27 sections, 4 theorems, 59 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. M-estimators under constraints:
  3. Stochastic approximation methods:
  4. Suboptimality of dual-averaging:
  5. Asymptotic optimality via Polyak-Ruppert averaging:
  6. Non-asymptotic bounds
  7. Contribution and organization
  8. An instance-dependent local minimax lower bound
  9. Perturbed problems and connections to local minimax lower bounds
  10. A particular perturbation of interest
  11. A non-asymptotic instance-dependent benchmark:
  12. Problem setup
  13. Main results
  14. Variance-reduced projected stochastic gradient
  15. Proximal step:
  16. ...and 12 more sections

Key Result

Proposition 1

Let $\widehat{x}_N$ be the output of any algorithm with access to $N$ iid samples $z_1, \ldots, z_N$. Then under suitable smoothness condition on the function $f$ and the constraint functions $g_i's$ in problem eqn:constraint-opt, we have the following asymptotic lower bound: where $Z \sim \mathcal{N}(0, \mathrm{P}_{\mathcal{T}} H^\star \mathrm{P}_{\mathcal{T}} \Sigma^\star \mathrm{P}_{\mathcal{T

Theorems & Definitions (4)

  • Proposition 1: Informal
  • Theorem 1
  • Lemma 1
  • Lemma 2