Stochastic Adaptive Optimization with Unreliable Inputs: A Unified Framework for High-Probability Complexity Analysis
Katya Scheinberg, Miaolan Xie
TL;DR
This work tackles unconstrained optimization of a differentiable, possibly non-convex objective $φ$ with a Lipschitz gradient, aiming to find an $ε$-stationary point, while access to $φ$ and $∇φ$ is mediated by probabilistic, unreliable oracles that may produce heavy-tailed or adversarially corrupted outputs. It introduces a unified adaptive optimization framework that encompasses trust-region and line-search strategies, and analyzes high-probability iteration complexity under very weak oracle assumptions (SZO, CZO, SFO). The main contributions are a general high-probability bound for the stopping time $T_ε$, specialized bounds for stochastic zeroth-order and corrupted zeroth-order inputs, and concrete instantiations for trust-region and line-search that yield $O(ε^{-2})$-type guarantees with explicit noise-driven convergence neighborhoods. This framework enables robust optimization in settings with data anomalies, such as corrupted datasets and heavy-tailed noise, and provides guidance on parameter choices to achieve reliable progress and convergence guarantees in practice.
Abstract
We consider an unconstrained continuous optimization problem where, in each iteration, gradient estimates may be arbitrarily corrupted with a probability greater than 1/2. Additionally, function value estimates may exhibit heavy-tailed noise. This setting captures challenging scenarios where both gradient and function value estimates can be unreliable, making it applicable to many real-world problems, which can have outliers and data anomalies. We introduce an algorithmic and analytical framework that provides high-probability bounds on iteration complexity for this setting. The analysis offers a unified approach, encompassing methods such as line search and trust region.