On the Dynamic Regret of Following the Regularized Leader: Optimism with History Pruning

Abstract

We revisit the Follow the Regularized Leader (FTRL) framework for Online Convex Optimization (OCO) over compact sets, focusing on achieving dynamic regret guarantees. Prior work has highlighted the framework's limitations in dynamic environments due to its tendency to produce "lazy" iterates. However, building on insights showing FTRL's ability to produce "agile" iterates, we show that it can indeed recover known dynamic regret bounds through optimistic composition of future costs and careful linearization of past costs, which can lead to pruning some of them. This new analysis of FTRL against dynamic comparators yields a principled way to interpolate between lazy and agile updates and offers several benefits, including refined control over regret terms, optimism without cyclic dependence, and the application of minimal recursive regularization akin to AdaFTRL. More broadly, we show that it is not the "lazy" projection style of FTRL that hinders (optimistic) dynamic regret, but the decoupling of the algorithm's state (linearized history) from its iterates, allowing the state to grow arbitrarily. Instead, pruning synchronizes these two when necessary.

Figures (2)

• Figure 1: Effect of dual state size on iterates agility. We consider two slots $t<t'$, where gradients switch direction: $\bm{g}_\tau\!=\!(-1, 0)$ for $\tau\leq t$, and $\bm{g}_\tau\!=\!(1, 0)$ for $\tau\!>\!t$. Top: Standard FTRL accumulates a large state $\bm{g}_{1:t}$, and the update via $\nabla r^\star(\bm{g}_{1:t})$ becomes insensitive to the change in direction; both $\bm{g}_{1:t}$ and $\bm{g}_{1:t'}$ map to the same iterate. Bottom: A well-maintained state $\bm{p}_{1:t}$ remains bounded, and hence its mapping stays close to $\mathcal{X}$, enabling $\bm{x}_{t'}$ to start aligning quickly with the new better iterate direction $(-1,0)$.
• Figure 2: Average dynamic regret over time across various non-stationary scenarios. Dashed lines indicate time slots where the computed iterate differs from the comparator $\bm{x}_t^\star$.

Theorems & Definitions (38)

• Theorem 3.1
• Theorem 3.2
• Theorem 3.3
• Theorem 3.4
• Lemma 4.1
• Lemma 4.2
• Lemma 4.3
• proof
• proof : Proof of Theorem \ref{['thm:1']}
• Lemma \ref{lemma:strong-dyn-opt-ftrl}