Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tight Rates for Bandit Control Beyond Quadratics

Y. Jennifer Sun, Zhou Lu

TL;DR

An algorithm is achieved that achieves an $\tilde{O}(\sqrt{T})$ optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known $\tilde{O}(T^{2/3})$ regret bound.

Abstract

Unlike classical control theory, such as Linear Quadratic Control (LQC), real-world control problems are highly complex. These problems often involve adversarial perturbations, bandit feedback models, and non-quadratic, adversarially chosen cost functions. A fundamental yet unresolved question is whether optimal regret can be achieved for these general control problems. The standard approach to addressing this problem involves a reduction to bandit convex optimization with memory. In the bandit setting, constructing a gradient estimator with low variance is challenging due to the memory structure and non-quadratic loss functions. In this paper, we provide an affirmative answer to this question. Our main contribution is an algorithm that achieves an $\tilde{O}(\sqrt{T})$ optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known $\tilde{O}(T^{2/3})$ regret bound from (Cassel and Koren, 2020. Our algorithm overcomes the memory issue by reducing the problem to Bandit Convex Optimization (BCO) without memory and addresses general strongly-convex costs using recent advancements in BCO from (Suggala et al., 2024). Along the way, we develop an improved algorithm for BCO with memory, which may be of independent interest.

Tight Rates for Bandit Control Beyond Quadratics

TL;DR

An algorithm is achieved that achieves an optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known regret bound.

Abstract

Unlike classical control theory, such as Linear Quadratic Control (LQC), real-world control problems are highly complex. These problems often involve adversarial perturbations, bandit feedback models, and non-quadratic, adversarially chosen cost functions. A fundamental yet unresolved question is whether optimal regret can be achieved for these general control problems. The standard approach to addressing this problem involves a reduction to bandit convex optimization with memory. In the bandit setting, constructing a gradient estimator with low variance is challenging due to the memory structure and non-quadratic loss functions. In this paper, we provide an affirmative answer to this question. Our main contribution is an algorithm that achieves an optimal regret for bandit non-stochastic control with strongly-convex and smooth cost functions in the presence of adversarial perturbations, improving the previously known regret bound from (Cassel and Koren, 2020. Our algorithm overcomes the memory issue by reducing the problem to Bandit Convex Optimization (BCO) without memory and addresses general strongly-convex costs using recent advancements in BCO from (Suggala et al., 2024). Along the way, we develop an improved algorithm for BCO with memory, which may be of independent interest.
Paper Structure (29 sections, 6 theorems, 76 equations, 1 table, 3 algorithms)

This paper contains 29 sections, 6 theorems, 76 equations, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Technical Overview
  3. Related Work
  4. Preliminary
  5. Online non-stochastic control with bandit feedback
  6. Bandit convex optimization with memory (BCO-M)
  7. Notations
  8. Improved Bandit Convex Optimization with Memory
  9. BCO-M algorithm
  10. Proof Sketch
  11. Step 1: regret of the base algorithm.
  12. Step 2: reduction to the base algorithm.
  13. Step 3: bounding the moving cost.
  14. Optimal Regret for Bandit Non-stochastic Control
  15. Conclusion
  16. ...and 14 more sections

Key Result

Theorem 6

Given an $(\alpha,\beta,G,D)$-well-conditioned BCO-M instance $\mathcal{O}=\{\mathcal{K}, m,\{f_t\}_{t\ge m, t\in\mathbb{N}}\}$ with $m=\mathrm{poly}(\log T)$ and $G,D=\tilde{O}(1)$ (def:bco-m), let $(\mathcal{K}, \eta=\Theta(1/\sqrt{T}), m, \alpha, T)$ be the input to alg:bco-am. alg:bco-am guarant where $\tilde{O}(\cdot)$ hides all universal constants and logarithmic dependence in $T$.

Theorems & Definitions (19)

  • Definition 1: DRC
  • Definition 2: Induced unary form
  • Definition 3: $\kappa_0$-convexity, suggala2024second
  • proof : Proof of \ref{['obs:unary-property']}
  • Definition 5: BCO-M instance
  • Theorem 6: BCO-M regret guarantee
  • Definition 7: Bandit non-stochastic control
  • Definition 8: Approximation
  • Lemma 9: Control reduction
  • Theorem 10: Bandit non-stochastic control regret guarantee
  • ...and 9 more