Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Decentralized Upper Confidence Bound Algorithms for Homogeneous Multi-Agent Multi-Armed Bandits

Jingxuan Zhu, Ethan Mulle, Christopher S. Smith, Alec Koppel, Ji Liu

TL;DR

The paper tackles decentralized cooperative multi-armed bandits with homogeneous rewards across agents by introducing fully decentralized KL-UCB and UCB1 algorithms that operate without any global network information. Through a consensus-like updating rule and a bifurcated decision process, each agent leverages neighbor information to achieve regret that scales inversely with its number of neighbors, outperforming isolated single-agent learning. Theoretical results provide asymptotic log-T regret bounds and, in some cases, best-possible scaling (e.g., O(log T / N) on complete graphs), while simulations validate the neighborhood-dependent gains and the superiority of KL-UCB over UCB1 in decentralized settings. The work further extends the framework to directed graphs and demonstrates substantial practical benefits for networked learning scenarios with privacy-preserving, fully decentralized communications.

Abstract

This paper studies a decentralized homogeneous multi-armed bandit problem in a multi-agent network. The problem is simultaneously solved by $N$ agents assuming they face a common set of $M$ arms and share the same arms' reward distributions. Each agent can receive information only from its neighbors, where the neighbor relationships among the agents are described by a fixed graph. Two fully decentralized upper confidence bound (UCB) algorithms are proposed for undirected graphs, respectively based on the classic algorithm and the state-of-the-art Kullback-Leibler upper confidence bound (KL-UCB) algorithm. The proposed decentralized UCB1 and KL-UCB algorithms permit each agent in the network to achieve a better logarithmic asymptotic regret than their single-agent counterparts, provided that the agent has at least one neighbor, and the more neighbors an agent has, the better regret it will have, meaning that the sum is more than its component parts. The same algorithm design framework is also extended to directed graphs through the design of a variant of the decentralized UCB1 algorithm, which outperforms the single-agent UCB1 algorithm.

Decentralized Upper Confidence Bound Algorithms for Homogeneous Multi-Agent Multi-Armed Bandits

TL;DR

The paper tackles decentralized cooperative multi-armed bandits with homogeneous rewards across agents by introducing fully decentralized KL-UCB and UCB1 algorithms that operate without any global network information. Through a consensus-like updating rule and a bifurcated decision process, each agent leverages neighbor information to achieve regret that scales inversely with its number of neighbors, outperforming isolated single-agent learning. Theoretical results provide asymptotic log-T regret bounds and, in some cases, best-possible scaling (e.g., O(log T / N) on complete graphs), while simulations validate the neighborhood-dependent gains and the superiority of KL-UCB over UCB1 in decentralized settings. The work further extends the framework to directed graphs and demonstrates substantial practical benefits for networked learning scenarios with privacy-preserving, fully decentralized communications.

Abstract

This paper studies a decentralized homogeneous multi-armed bandit problem in a multi-agent network. The problem is simultaneously solved by agents assuming they face a common set of arms and share the same arms' reward distributions. Each agent can receive information only from its neighbors, where the neighbor relationships among the agents are described by a fixed graph. Two fully decentralized upper confidence bound (UCB) algorithms are proposed for undirected graphs, respectively based on the classic algorithm and the state-of-the-art Kullback-Leibler upper confidence bound (KL-UCB) algorithm. The proposed decentralized UCB1 and KL-UCB algorithms permit each agent in the network to achieve a better logarithmic asymptotic regret than their single-agent counterparts, provided that the agent has at least one neighbor, and the more neighbors an agent has, the better regret it will have, meaning that the sum is more than its component parts. The same algorithm design framework is also extended to directed graphs through the design of a variant of the decentralized UCB1 algorithm, which outperforms the single-agent UCB1 algorithm.
Paper Structure (11 sections, 13 theorems, 120 equations, 13 figures)

This paper contains 11 sections, 13 theorems, 120 equations, 13 figures.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Algorithms
  4. Decentralized KL-UCB
  5. Results
  6. Decentralized UCB1
  7. Results
  8. Analysis
  9. Simulations
  10. Additional Simulations
  11. Conclusion

Key Result

Theorem 1

Suppose that $\mathbb G$ is undirected and connected, and that all $N$ agents adhere to Algorithm algorithm:KL. Then, with bounded rewards over $[0,1]$, Metropolis weights $w_{ij}$ given in eq:metroweights, and where $\varsigma_i$ is an arbitrary positive constant, the regret of each agent $i$ until time $T$ satisfies where $\Psi(\epsilon, T)$ is a positive function satisfying $\Psi(\epsilon, T)

Figures (13)

  • Figure 1: UCB1 vs. decentralized UCB1 over directed graphs
  • Figure 2: UCB1 vs. decentralized UCB1
  • Figure 3: KL-UCB vs. decentralized KL-UCB
  • Figure 4: Plots of the mean regrets of decentralized KL-UCB vs. decentralized UCB1 for $\varsigma=\beta=0.01$
  • Figure 5: Plots of the mean regrets of decentralized UCB1 in our paper vs. UCB1 algorithm in jingxuan2021.
  • ...and 8 more figures

Theorems & Definitions (18)

  • Theorem 1
  • Remark 1
  • Remark 2
  • Corollary 1
  • Remark 3
  • Corollary 2
  • Theorem 2
  • Remark 4
  • Corollary 3
  • Theorem 3
  • ...and 8 more