First Order Stochastic Optimization with Oblivious Noise

TL;DR

This work introduces and analyzes stochastic optimization under oblivious noise, where the gradient oracle returns $\nabla f(\gamma,x) + \xi$ with $\Pr[\xi=0]\ge \alpha$, potentially allowing extreme tails. It shows that exact recovery is information-theoretically impossible for $\alpha\le 1/2$, and proposes a principled list-decodable framework that yields a small set of candidate solutions with at least one near-optimal member; in the regime $\alpha=1-\varepsilon$ it can recover a single solution. The core methodological pillar is an equivalence between list-decodable stochastic optimization (LDSO) and list-decodable mean estimation (LDME), supported by a finite-sample noisy location-estimation technique based on rejection sampling. The paper provides two main reductions: LDSO reduces to LDME and LDME reduces to LDSO, each with precise complexity bounds, and an extension to higher dimensions via random rotations. Collectively, these results establish a robust framework for first-order optimization under heavy, even adversarially corrupted, noise with provable guarantees and practical location-estimation tools, while highlighting fundamental limits and directions for improving dependence on the precision parameter.

Abstract

We initiate the study of stochastic optimization with oblivious noise, broadly generalizing the standard heavy-tailed noise setup. In our setting, in addition to random observation noise, the stochastic gradient may be subject to independent oblivious noise, which may not have bounded moments and is not necessarily centered. Specifically, we assume access to a noisy oracle for the stochastic gradient of $f$ at $x$, which returns a vector $\nabla f(γ, x) + ξ$, where $γ$ is the bounded variance observation noise and $ξ$ is the oblivious noise that is independent of $γ$ and $x$. The only assumption we make on the oblivious noise $ξ$ is that $\mathbf{Pr}[ξ= 0] \ge α$ for some $α\in (0, 1)$. In this setting, it is not information-theoretically possible to recover a single solution close to the target when the fraction of inliers $α$ is less than $1/2$. Our main result is an efficient list-decodable learner that recovers a small list of candidates, at least one of which is close to the true solution. On the other hand, if $α= 1-ε$, where $0< ε< 1/2$ is sufficiently small constant, the algorithm recovers a single solution. Along the way, we develop a rejection-sampling-based algorithm to perform noisy location estimation, which may be of independent interest.

Figures (1)

• Figure 1: An illustration showing why the median of $x + y$ can deviate significantly, even if $x$ is symmetric and $y$ has a standard deviation of $O(1)$. Here, $x$ (shown in blue) is symmetric around the origin, rarely $0$, and typically $\pm 5$. Conversely, $y$ (shown in orange) is highly skewed with mean $0$ and low variance. While both $x$ and $y$ have medians near $0$, the median of $x + y$ (shown in green) is near $5$.

Theorems & Definitions (40)

• Definition 1.1: Oblivious Noise Oracle
• Definition 1.2: List-Decodable Stochastic Optimization
• Definition 1.3: List-Decodable Mean Estimation
• Theorem 1.4: List-Decodable Stochastic Optimization $\rightarrow$ List-Decodable Mean Estimation
• Theorem 1.5: Noisy Location Estimation
• Theorem 1.6: List-Decodable Mean Estimation $\rightarrow$ List-Decodable Stochastic Optimization
• Definition 2.2: Robust Inexact Learner
• Lemma 3.1: One-dimensional location-estimation
• proof
• Claim 3.2: Rough Estimate