Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improved Regret for Bandit Convex Optimization with Delayed Feedback

Yuanyu Wan, Chang Yao, Mingli Song, Lijun Zhang

TL;DR

This work addresses bandit convex optimization under delayed feedback, where only delayed loss values are observed. It introduces the delayed follow-the-bandit-leader (D-FTBL) algorithm, which uses a blocking update mechanism to decouple the joint effects of delays and bandit feedback, and analyzes its regret for convex and strongly convex settings. For convex losses, it achieves $O(\sqrt{n}T^{3/4}+\sqrt{dT})$, and for strongly convex losses, $O((nT)^{2/3}\log^{1/3}T + d\log T)$, with an unconstrained-extension giving $O(n\sqrt{T\log T}+d\log T)$ in the smooth case. These results tighten the delay-dependent term and match lower bounds in the worst case, advancing the understanding of delayed feedback in BCO and offering practical improvements when delays are large.

Abstract

We investigate bandit convex optimization (BCO) with delayed feedback, where only the loss value of the action is revealed under an arbitrary delay. Let $n,T,\bar{d}$ denote the dimensionality, time horizon, and average delay, respectively. Previous studies have achieved an $O(\sqrt{n}T^{3/4}+(n\bar{d})^{1/3}T^{2/3})$ regret bound for this problem, whose delay-independent part matches the regret of the classical non-delayed bandit gradient descent algorithm. However, there is a large gap between its delay-dependent part, i.e., $O((n\bar{d})^{1/3}T^{2/3})$, and an existing $Ω(\sqrt{\bar{d}T})$ lower bound. In this paper, we illustrate that this gap can be filled in the worst case, where $\bar{d}$ is very close to the maximum delay $d$. Specifically, we first develop a novel algorithm, and prove that it enjoys a regret bound of $O(\sqrt{n}T^{3/4}+\sqrt{dT})$ in general. Compared with the previous result, our regret bound is better for $d=O((n\bar{d})^{2/3}T^{1/3})$, and the delay-dependent part is tight in the worst case. The primary idea is to decouple the joint effect of the delays and the bandit feedback on the regret by carefully incorporating the delayed bandit feedback with a blocking update mechanism. Furthermore, we show that the proposed algorithm can improve the regret bound to $O((nT)^{2/3}\log^{1/3}T+d\log T)$ for strongly convex functions. Finally, if the action sets are unconstrained, we demonstrate that it can be simply extended to achieve an $O(n\sqrt{T\log T}+d\log T)$ regret bound for strongly convex and smooth functions.

Improved Regret for Bandit Convex Optimization with Delayed Feedback

TL;DR

This work addresses bandit convex optimization under delayed feedback, where only delayed loss values are observed. It introduces the delayed follow-the-bandit-leader (D-FTBL) algorithm, which uses a blocking update mechanism to decouple the joint effects of delays and bandit feedback, and analyzes its regret for convex and strongly convex settings. For convex losses, it achieves , and for strongly convex losses, , with an unconstrained-extension giving in the smooth case. These results tighten the delay-dependent term and match lower bounds in the worst case, advancing the understanding of delayed feedback in BCO and offering practical improvements when delays are large.

Abstract

We investigate bandit convex optimization (BCO) with delayed feedback, where only the loss value of the action is revealed under an arbitrary delay. Let denote the dimensionality, time horizon, and average delay, respectively. Previous studies have achieved an regret bound for this problem, whose delay-independent part matches the regret of the classical non-delayed bandit gradient descent algorithm. However, there is a large gap between its delay-dependent part, i.e., , and an existing lower bound. In this paper, we illustrate that this gap can be filled in the worst case, where is very close to the maximum delay . Specifically, we first develop a novel algorithm, and prove that it enjoys a regret bound of in general. Compared with the previous result, our regret bound is better for , and the delay-dependent part is tight in the worst case. The primary idea is to decouple the joint effect of the delays and the bandit feedback on the regret by carefully incorporating the delayed bandit feedback with a blocking update mechanism. Furthermore, we show that the proposed algorithm can improve the regret bound to for strongly convex functions. Finally, if the action sets are unconstrained, we demonstrate that it can be simply extended to achieve an regret bound for strongly convex and smooth functions.
Paper Structure (16 sections, 10 theorems, 57 equations, 1 algorithm)

This paper contains 16 sections, 10 theorems, 57 equations, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Standard OCO and BCO
  4. OCO and BCO with delays
  5. Main results
  6. Preliminaries
  7. Our improved algorithm
  8. Theoretical guarantees
  9. Analysis
  10. Proof of Theorem \ref{['thm1']}
  11. Conclusion and future work
  12. Proof of Lemma \ref{['pre_thm1']}
  13. Proof of Lemmas \ref{['block-gradient']} and \ref{['delay-block-gradient']}
  14. Proof of Theorem \ref{['thm2']}
  15. Proof of Theorem \ref{['thm2-1']}
  16. ...and 1 more sections

Key Result

Lemma 1

(Lemma 1 in OBO05) Given a function $f(\mathbf{x}):\mathbb{R}^n\mapsto\mathbb{R}$ and a constant $\delta\in(0,1)$, its $\delta$-smoothed version $\hat{f}_\delta(\mathbf{x})$ defined in delta-smooth ensures where $\mathcal{S}^n$ denotes the unit Euclidean sphere centered at the origin in $\mathbb{R}^n$.

Theorems & Definitions (12)

  • Definition 1
  • Definition 2
  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • ...and 2 more