Making SGD Parameter-Free
Yair Carmon, Oliver Hinder
TL;DR
This work delivers a parameter-free approach to stochastic convex optimization by developing a computable SGD step-size certificate that replaces dependence on the unknown initial distance to optimum with the observable iterate excursion. The core idea uses a time-uniform, data-adaptive criterion and a doubling-bisection procedure to find a near-optimal step size, enabling high-probability convergence guarantees that are only a double-logarithmic factor worse than the known-parameter optimum. Beyond exact gradients, the authors extend the framework to stochastic gradients via a good-event concentration analysis, achieving parameter-free SCO rates that adapt to gradient norms, smoothness, and, through restarts, to strong convexity. The approach yields the first high-probability, parameter-free rates for SCO, improving over regret-based parameter-free methods by removing logarithmic penalties and providing localization guarantees that keep iterates close to the optimum. The practical impact lies in robust SGD tuning without prior problem knowledge, with adaptive behavior to problem structure and a simple, implementable procedure based on a bisection over step sizes.
Abstract
We develop an algorithm for parameter-free stochastic convex optimization (SCO) whose rate of convergence is only a double-logarithmic factor larger than the optimal rate for the corresponding known-parameter setting. In contrast, the best previously known rates for parameter-free SCO are based on online parameter-free regret bounds, which contain unavoidable excess logarithmic terms compared to their known-parameter counterparts. Our algorithm is conceptually simple, has high-probability guarantees, and is also partially adaptive to unknown gradient norms, smoothness, and strong convexity. At the heart of our results is a novel parameter-free certificate for SGD step size choice, and a time-uniform concentration result that assumes no a-priori bounds on SGD iterates.