Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Information Capacity Regret Bounds for Bandits with Mediator Feedback

Khaled Eldowa, Nicolò Cesa-Bianchi, Alberto Maria Metelli, Marcello Restelli

TL;DR

The paper studies bandits with mediator feedback where each policy induces a distribution over outcomes and the learner observes a sampled outcome and its loss. It introduces the policy-set capacity based on chi-squared information, and shows EXP4 regret bounds that scale with this capacity in adversarial and time-varying settings, plus near-matching lower bounds for several policy families. It also establishes best-of-both-worlds bounds in stochastic/adversarial regimes, proves an impossibility result separating mediator feedback from linear bandits, and analyzes the full-information variant via KL-based capacity. These results reveal that exploiting policy similarity through the capacity measure can substantially improve regret guarantees and delineate fundamental limits of feedback models in bandit problems.

Abstract

This work addresses the mediator feedback problem, a bandit game where the decision set consists of a number of policies, each associated with a probability distribution over a common space of outcomes. Upon choosing a policy, the learner observes an outcome sampled from its distribution and incurs the loss assigned to this outcome in the present round. We introduce the policy set capacity as an information-theoretic measure for the complexity of the policy set. Adopting the classical EXP4 algorithm, we provide new regret bounds depending on the policy set capacity in both the adversarial and the stochastic settings. For a selection of policy set families, we prove nearly-matching lower bounds, scaling similarly with the capacity. We also consider the case when the policies' distributions can vary between rounds, thus addressing the related bandits with expert advice problem, which we improve upon its prior results. Additionally, we prove a lower bound showing that exploiting the similarity between the policies is not possible in general under linear bandit feedback. Finally, for a full-information variant, we provide a regret bound scaling with the information radius of the policy set.

Information Capacity Regret Bounds for Bandits with Mediator Feedback

TL;DR

The paper studies bandits with mediator feedback where each policy induces a distribution over outcomes and the learner observes a sampled outcome and its loss. It introduces the policy-set capacity based on chi-squared information, and shows EXP4 regret bounds that scale with this capacity in adversarial and time-varying settings, plus near-matching lower bounds for several policy families. It also establishes best-of-both-worlds bounds in stochastic/adversarial regimes, proves an impossibility result separating mediator feedback from linear bandits, and analyzes the full-information variant via KL-based capacity. These results reveal that exploiting policy similarity through the capacity measure can substantially improve regret guarantees and delineate fundamental limits of feedback models in bandit problems.

Abstract

This work addresses the mediator feedback problem, a bandit game where the decision set consists of a number of policies, each associated with a probability distribution over a common space of outcomes. Upon choosing a policy, the learner observes an outcome sampled from its distribution and incurs the loss assigned to this outcome in the present round. We introduce the policy set capacity as an information-theoretic measure for the complexity of the policy set. Adopting the classical EXP4 algorithm, we provide new regret bounds depending on the policy set capacity in both the adversarial and the stochastic settings. For a selection of policy set families, we prove nearly-matching lower bounds, scaling similarly with the capacity. We also consider the case when the policies' distributions can vary between rounds, thus addressing the related bandits with expert advice problem, which we improve upon its prior results. Additionally, we prove a lower bound showing that exploiting the similarity between the policies is not possible in general under linear bandit feedback. Finally, for a full-information variant, we provide a regret bound scaling with the information radius of the policy set.
Paper Structure (25 sections, 12 theorems, 100 equations, 3 algorithms)

This paper contains 25 sections, 12 theorems, 100 equations, 3 algorithms.

Table of Contents

  1. Introduction
  2. Prior Results
  3. Contributions
  4. Additional Related Works
  5. Preliminaries
  6. Information Theory Background
  7. Problem Setting
  8. The Policy Set Capacity: An Improved Regret Bound for EXP4
  9. The Policy Set Capacity
  10. A Regret Bound for EXP4 in Terms of the Capacity
  11. Examples
  12. Two Policies
  13. Epsilon-Greedy Policies
  14. M-Supported Uniform Policies
  15. A Generalization for Time-Varying Policy Distributions
  16. ...and 10 more sections

Key Result

Theorem 1

Algorithm alg:exp4 with $\eta_t = \min\Bigl\{1,\sqrt{ \frac{\log N}{e \mathcal{C}(\Theta) t }}\Bigr\}$ satisfies

Theorems & Definitions (12)

  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 4
  • Theorem 5
  • Theorem 6
  • Theorem 7
  • Proposition 8
  • Theorem 9
  • Theorem 10
  • ...and 2 more