Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Best Arm Identification in Stochastic Linear Bandits

Sanjana Agrawal, Saúl A. Blanco

TL;DR

This work tackles collaborative fixed-budget best-arm identification in stochastic linear bandits with multiple agents connected via a network. It introduces MaLinBAI-Star for star networks and MaLinBAI-Gen for generic networks, both leveraging G-optimal design and a successive-elimination strategy guided by a central server, with MaLinBAI-Gen exploiting dominating-set partitions to reduce communication. The authors prove exponential decay of the error probability in the time budget and show near-optimality relative to known lower bounds, while achieving efficient communication costs, especially in generic networks. Empirical results on synthetic and real-world data demonstrate strong accuracy improvements over several baselines and favorable communication efficiency, validating the proposed approaches for federated linear-bandit pure exploration.

Abstract

We study the problem of collaborative best-arm identification in stochastic linear bandits under a fixed-budget scenario. In our learning model, we first consider multiple agents connected through a star network, interacting with a linear bandit instance in parallel. We then extend our analysis to arbitrary network topologies. The objective of the agents is to collaboratively identify the best arm of the given bandit instance with the help of a central server while minimizing the probability of error in best arm estimation. To this end, we propose two algorithms, MaLinBAI-Star and MaLinBAI-Gen for star networks and networks with arbitrary structure, respectively. Both algorithms utilize the technique of G-optimal design along with the successive elimination based strategy where agents share their knowledge through a central server at each communication round. We demonstrate, both theoretically and empirically, that our algorithms achieve exponentially decaying probability of error in the allocated time budget. Furthermore, experimental results on both synthetic and real-world data validate the effectiveness of our algorithms over the state-of-the art existing multi-agent algorithms.

Multi-Agent Best Arm Identification in Stochastic Linear Bandits

TL;DR

This work tackles collaborative fixed-budget best-arm identification in stochastic linear bandits with multiple agents connected via a network. It introduces MaLinBAI-Star for star networks and MaLinBAI-Gen for generic networks, both leveraging G-optimal design and a successive-elimination strategy guided by a central server, with MaLinBAI-Gen exploiting dominating-set partitions to reduce communication. The authors prove exponential decay of the error probability in the time budget and show near-optimality relative to known lower bounds, while achieving efficient communication costs, especially in generic networks. Empirical results on synthetic and real-world data demonstrate strong accuracy improvements over several baselines and favorable communication efficiency, validating the proposed approaches for federated linear-bandit pure exploration.

Abstract

We study the problem of collaborative best-arm identification in stochastic linear bandits under a fixed-budget scenario. In our learning model, we first consider multiple agents connected through a star network, interacting with a linear bandit instance in parallel. We then extend our analysis to arbitrary network topologies. The objective of the agents is to collaboratively identify the best arm of the given bandit instance with the help of a central server while minimizing the probability of error in best arm estimation. To this end, we propose two algorithms, MaLinBAI-Star and MaLinBAI-Gen for star networks and networks with arbitrary structure, respectively. Both algorithms utilize the technique of G-optimal design along with the successive elimination based strategy where agents share their knowledge through a central server at each communication round. We demonstrate, both theoretically and empirically, that our algorithms achieve exponentially decaying probability of error in the allocated time budget. Furthermore, experimental results on both synthetic and real-world data validate the effectiveness of our algorithms over the state-of-the art existing multi-agent algorithms.

Paper Structure

This paper contains 19 sections, 11 theorems, 31 equations, 8 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Preliminaries
  4. Algorithm for Star Networks
  5. Algorithm for Generic Networks
  6. Experiments
  7. Conclusion and Future Work
  8. Notations and Technical Lemmas
  9. Missing proofs of Section \ref{['section:star']}
  10. Proof of Lemma \ref{['lemma1']}
  11. Proof of Lemma \ref{['lemma2']}
  12. Missing proofs of Section \ref{['section:gen']}
  13. Proof of Theorem \ref{['theorem2']}
  14. Proof of Lower Bound
  15. Lower Bound for Error Probability
  16. ...and 4 more sections

Key Result

Lemma 1

Let $a_1$ be the best arm, and suppose that $a_1$ is not eliminated prior to round $p$, the probability that any suboptimal arm $a_i$ has estimated expected reward higher than that of $a_1$ during round $p$ of MaLinBAI-Star is given by

Figures (8)

  • Figure 1: Example graph with 8 nodes
  • Figure 2: Corresponding 2-tier star network structure from the dominating set partition $\{P_1, P_2\} = \{\{1,2,5,6\},\{3,4,7,8\}\}$ arising from the dominating set $\{1,4\}$.
  • Figure 4: Standard instance
  • Figure 5: Random instance
  • Figure 7: Standard instance
  • ...and 3 more figures

Theorems & Definitions (16)

  • Lemma 1
  • Lemma 2
  • Theorem 1
  • proof
  • Theorem 2
  • Lemma 3: Kiefer and Wolfowitzkiefer1960equivalence
  • Lemma 4: Lattimore and Szepesvárilattimore2020bandit
  • Lemma 5: Markov's Inequality
  • Lemma 6
  • proof
  • ...and 6 more