A Reduction from Delayed to Immediate Feedback for Online Convex Optimization with Improved Guarantees

TL;DR

A continuous-time model under which regret decomposes into a delay-independent learning term and a delay-induced drift term is introduced, yielding a delay-adaptive reduction that converts any algorithm for online linear optimization into one that handles round-dependent delays.

Abstract

We develop a reduction-based framework for online learning with delayed feedback that recovers and improves upon existing results for both first-order and bandit convex optimization. Our approach introduces a continuous-time model under which regret decomposes into a delay-independent learning term and a delay-induced drift term, yielding a delay-adaptive reduction that converts any algorithm for online linear optimization into one that handles round-dependent delays. For bandit convex optimization, we significantly improve existing regret bounds, with delay-dependent terms matching state-of-the-art first-order rates. For first-order feedback, we recover state-of-the-art regret bounds via a simpler, unified analysis. Quantitatively, for bandit convex optimization we obtain $O(\sqrt{d_{\text{tot}}} + T^{\frac{3}{4}}\sqrt{k})$ regret, improving the delay-dependent term from $O(\min\{\sqrt{T d_{\text{max}}},(Td_{\text{tot}})^{\frac{1}{3}}\})$ in previous work to $O(\sqrt{d_{\text{tot}}})$. Here, $k$, $T$, $d_{\text{max}}$, and $d_{\text{tot}}$ denote the dimension, time horizon, maximum delay, and total delay, respectively. Under strong convexity, we achieve $O(\min\{σ_{\text{max}} \ln T, \sqrt{d_{\text{tot}}}\} + (T^2\ln T)^{\frac{1}{3}} {k}^{\frac{2}{3}})$, improving the delay-dependent term from $O(d_{\text{max}} \ln T)$ in previous work to $O(\min\{σ_{\text{max}} \ln T, \sqrt{d_{\text{tot}}}\})$, where $σ_{\text{max}}$ denotes the maximum number of outstanding observations and may be considerably smaller than $d_{\text{max}}$.

Figures (6)

• Figure 1: OCO with delays under first-order or bandit feedback.
• Figure 2: Example of latency-intervals for $T\!=\!5$. The table provides values $d_t, \sigma_t, d^{\star}_t, \sigma^{\star}_t$ and $\beta:[T]\to[T]$ from Theorem \ref{['thm:CTM']}.
• Figure 3: Partition of rounds induced by the latency intervals in Fig. \ref{['fig:illustration']}: if $t\in Z_n$ (equivalently, $l_t\in[\widetilde{r}_{n-1},\widetilde{r}_n)$), then a steady algorithm outputs $x_t=z_n$. The table provides corresponding values $\widetilde{d}_n, \widetilde{\sigma}_n, \widetilde{d}^{\star}_n, \widetilde{\sigma}^{\star}_n$ and permutation $\rho:[T]\to[T]$.
• Figure 4: Regret decomposition for steady algorithms.
• Figure 5: Online learning with drift penalization for linear losses.

Theorems & Definitions (83)

• Remark 2.6: Differentiability of loss functions
• Definition 2.7: Steady Algorithm
• Definition 3.1: Continuous Time Model (CTM)
• Lemma 3.2
• proof
• Remark 3.3: Consistency and Realizability in the CTM
• Theorem 3.4
• Definition 3.5: Observation-Ordering
• Lemma 3.6
• proof