Dealing with unbounded gradients in stochastic saddle-point optimization
Gergely Neu, Nneka Okolo
TL;DR
This work addresses instability in stochastic saddle-point optimization caused by unbounded gradients by introducing a simple regularization-based stabilization that removes the need for projection radii. The core technique augments the objective with initialization-centered penalties, yielding initialization-adaptive duality-gap guarantees that remain valid under data-dependent comparators and even multiplicative gradient noise. The framework extends from unconstrained bilinear games to sub-bilinear objectives with general Bregman divergences and applies to complex domains, including Average-Reward MDP planning via COMIDA-MDP, achieving near-optimal policies without prior bounds on value functions. These results offer a parameter-free, projection-free approach to robust saddle-point optimization with broad applicability and practical reinforcement learning implications.
Abstract
We study the performance of stochastic first-order methods for finding saddle points of convex-concave functions. A notorious challenge faced by such methods is that the gradients can grow arbitrarily large during optimization, which may result in instability and divergence. In this paper, we propose a simple and effective regularization technique that stabilizes the iterates and yields meaningful performance guarantees even if the domain and the gradient noise scales linearly with the size of the iterates (and is thus potentially unbounded). Besides providing a set of general results, we also apply our algorithm to a specific problem in reinforcement learning, where it leads to performance guarantees for finding near-optimal policies in an average-reward MDP without prior knowledge of the bias span.