Metric Entropy-Free Sample Complexity Bounds for Sample Average Approximation in Convex Stochastic Programming

TL;DR

The paper addresses the non-asymptotic sample complexity of SAA for convex and strongly convex stochastic programs by deriving metric entropy-free bounds under standard assumptions, removing reliance on the feasible region's metric entropy. It shows that SAA can match the sample efficiency of stochastic mirror descent (SMD) in both strongly convex and convex cases, even with heavy-tailed subgradients, and extends these results to non-Lipschitz settings with moment-based conditions. A second set of results provides large-deviation-type bounds without metric entropy terms under Lipschitz/light-tail conditions, yielding dimension-insensitive rates up to poly-log factors in the confidence level β. Numerical experiments corroborate the theory, demonstrating that regularized SAA variants with Tikhonov-type penalties can outperform unregularized SAA and rival LASSO in high dimensions, while maintaining comparable computational effort. Overall, the work substantially narrows the theoretical gap between SAA and SMD and broadens the practical applicability of SAA in irregular SP problems.

Abstract

This paper studies sample average approximation (SAA) in solving convex or strongly convex stochastic programming (SP) problems. In estimating SAA's sample efficiency, the state-of-the-art sample complexity bounds entail metric entropy terms (such as the logarithm of the feasible region's covering number), which often grow polynomially with problem dimensionality. While it has been shown that metric entropy-free complexity rates are attainable under a uniform Lipschitz condition, such an assumption can be overly critical for many important SP problem settings. In response, this paper presents perhaps the first set of metric entropy-free sample complexity bounds for the SAA under standard SP assumptions -- in the absence of the uniform Lipschitz condition. The new results often lead to an $O(d)$-improvement in the complexity rate than the state-of-the-art. From the newly established complexity bounds, an important revelation is that SAA and the canonical stochastic mirror descent (SMD) method, two mainstream solution approaches to SP, entail almost identical rates of sample efficiency, lifting a theoretical discrepancy of SAA from SMD also by the order of $O(d)$. Furthermore, this paper explores non-Lipschitzian scenarios where SAA maintains provable efficacy but the corresponding results for SMD remain mostly unexplored, indicating the potential of SAA's better applicability in some irregular settings. Our numerical experiment results on SAA for solving a simulated SP problem align with our theoretical findings.

Figures (2)

• Figure 1: Averge suboptimality gaps (over five replications) achieved by different variations of SAA and their comparison with LASSO when dimensionality increases. The three different rows of subplots present results for sample sizes $N$ of 200, 400, and 600, respectively. The subplots on the bottom row (namely, subplots (a.2), (b.2), and (c.2)) present magnified views of the ones on the top row (Suplots (a.1), (b.1) and (c.1), respectively) without displaying "SAA$_r$"
• Figure 2: Averge computational time (s) over five replications achieved by different variations of SAA and their comparison with LASSO when dimensionality increases. The subplots in the top, middle, and bottom report results for $N=200$, $400$, and $600$, respectively.

Theorems & Definitions (32)

• remark thmcounterremark
• theorem 1
• proof
• remark thmcounterremark
• remark thmcounterremark
• remark thmcounterremark
• theorem 2
• proof
• remark thmcounterremark
• remark thmcounterremark