Partially Lazy Gradient Descent for Smoothed Online Learning

TL;DR

The paper tackles dynamic regret in Smoothed Online Convex Optimization by introducing k-lazyGD, a partially lazy gradient method that interpolates between greedy updates ($k=1$) and fully lazy dual-averaging ($k=T$). It casts the algorithm as an FTRL instance with a principled pruning rule, proving equivalence to a phased projection update and deriving both upper and lower bounds. The main results show that for laziness up to $k^*= ext{Θ}( ext{√}(T/P_T))$, the method achieves the minimax dynamic regret $ ext{R}_T= ext{O}( ext{√}((P_T+1)T))$, with a matching lower bound indicating this is tight. To handle unknown comparator non-stationarity, the authors deploy an ensemble of $k$-lazyGD learners with meta-learning to adaptively select effective slacks, yielding stable yet agile updates in practice. Overall, the work unifies stability and responsiveness in SOCO through a tunable laziness parameter and an adaptive ensemble, offering both theoretical optimality and practical adaptability.

Abstract

We introduce $k$-lazyGD, an online learning algorithm that bridges the gap between greedy Online Gradient Descent (OGD, for $k=1$) and lazy GD/dual-averaging (for $k=T$), creating a spectrum between reactive and stable updates. We analyze this spectrum in Smoothed Online Convex Optimization (SOCO), where the learner incurs both hitting and movement costs. Our main contribution is establishing that laziness is possible without sacrificing hitting performance: we prove that $k$-lazyGD achieves the optimal dynamic regret $\mathcal{O}(\sqrt{(P_T+1)T})$ for any laziness slack $k$ up to $Θ(\sqrt{T/P_T})$, where $P_T$ is the comparator path length. This result formally connects the allowable laziness to the comparator's shifts, showing that $k$-lazyGD can retain the inherently small movements of lazy methods without compromising tracking ability. We base our analysis on the Follow the Regularized Leader (FTRL) framework, and derive a matching lower bound. Since the slack depends on $P_T$, an ensemble of learners with various slacks is used, yielding a method that is provably stable when it can be, and agile when it must be.

Figures (11)

• Figure 1: Switching in example ($i$, top), showing staleness, and example ($ii$, bottom), showing stability. Left: switching cost. Right: Snapshots over 4 (top) and 2 (bottom) rounds: greedy updates move continuously, whereas lazy updates remain still or move minimally.
• Figure 2: The constructed sequence ($k=4$). $g_t^I$ is zero within blocks by \ref{['eq:pruning-cond-n']}, and at block starts since $y=0$ then.
• Figure 3: Geometric intuition for the effect of lazy vs. greedy updates in $\ell_1$ (left) and $\ell_2$ (right) ball domains. From $y$, greedy projects $y-g_t$, while lazy projects $y-g_t-g_{t-1}$ ($k=2$, $\sigma=1$). Blue dots are projections.
• Figure 4: Switching and hitting cost in Example $(i)$.
• Figure 5: Switching and hitting cost in Example $(ii)$.

Theorems & Definitions (25)

• Theorem 1
• proof
• Theorem 2
• proof
• Proposition 3: Staleness of Lazy Iterates
• proof
• Proposition 4: Stability of Lazy Iterates
• proof
• Theorem 5
• Lemma 6