Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Agent Bandit Learning through Heterogeneous Action Erasure Channels

Osama A. Hanna, Merve Karakas, Lin F. Yang, Christina Fragouli

TL;DR

This paper tackles multi‑agent MAB with heterogeneous action erasures and no feedback by introducing BatchSP2, a batched SAE‑based algorithm that uses per‑agent repetition and a sophisticated scheduling strategy to counter erasures. The authors prove sub‑linear regret bounds that account for channel heterogeneity and provide both instance‑dependent and gap‑dependent guarantees. Empirical results compare BatchSP2 to multiple baselines and show superior performance, especially as erasure probabilities vary across agents. The work advances learning under communication constraints in distributed systems and has practical implications for scalable, robust decision making with limited feedback channels.

Abstract

Multi-Armed Bandit (MAB) systems are witnessing an upswing in applications within multi-agent distributed environments, leading to the advancement of collaborative MAB algorithms. In such settings, communication between agents executing actions and the primary learner making decisions can hinder the learning process. A prevalent challenge in distributed learning is action erasure, often induced by communication delays and/or channel noise. This results in agents possibly not receiving the intended action from the learner, subsequently leading to misguided feedback. In this paper, we introduce novel algorithms that enable learners to interact concurrently with distributed agents across heterogeneous action erasure channels with different action erasure probabilities. We illustrate that, in contrast to existing bandit algorithms, which experience linear regret, our algorithms assure sub-linear regret guarantees. Our proposed solutions are founded on a meticulously crafted repetition protocol and scheduling of learning across heterogeneous channels. To our knowledge, these are the first algorithms capable of effectively learning through heterogeneous action erasure channels. We substantiate the superior performance of our algorithm through numerical experiments, emphasizing their practical significance in addressing issues related to communication constraints and delays in multi-agent environments.

Multi-Agent Bandit Learning through Heterogeneous Action Erasure Channels

TL;DR

This paper tackles multi‑agent MAB with heterogeneous action erasures and no feedback by introducing BatchSP2, a batched SAE‑based algorithm that uses per‑agent repetition and a sophisticated scheduling strategy to counter erasures. The authors prove sub‑linear regret bounds that account for channel heterogeneity and provide both instance‑dependent and gap‑dependent guarantees. Empirical results compare BatchSP2 to multiple baselines and show superior performance, especially as erasure probabilities vary across agents. The work advances learning under communication constraints in distributed systems and has practical implications for scalable, robust decision making with limited feedback channels.

Abstract

Multi-Armed Bandit (MAB) systems are witnessing an upswing in applications within multi-agent distributed environments, leading to the advancement of collaborative MAB algorithms. In such settings, communication between agents executing actions and the primary learner making decisions can hinder the learning process. A prevalent challenge in distributed learning is action erasure, often induced by communication delays and/or channel noise. This results in agents possibly not receiving the intended action from the learner, subsequently leading to misguided feedback. In this paper, we introduce novel algorithms that enable learners to interact concurrently with distributed agents across heterogeneous action erasure channels with different action erasure probabilities. We illustrate that, in contrast to existing bandit algorithms, which experience linear regret, our algorithms assure sub-linear regret guarantees. Our proposed solutions are founded on a meticulously crafted repetition protocol and scheduling of learning across heterogeneous channels. To our knowledge, these are the first algorithms capable of effectively learning through heterogeneous action erasure channels. We substantiate the superior performance of our algorithm through numerical experiments, emphasizing their practical significance in addressing issues related to communication constraints and delays in multi-agent environments.
Paper Structure (17 sections, 6 theorems, 35 equations, 2 figures, 2 tables, 2 algorithms)

This paper contains 17 sections, 6 theorems, 35 equations, 2 figures, 2 tables, 2 algorithms.

Table of Contents

  1. INTRODUCTION
  2. Related Work.
  3. Paper Organization
  4. PROBLEM FORMULATION
  5. Multi-armed Bandits
  6. Multi-Agent Multi-Armed Bandits with Action Erasures
  7. PROPOSED ALGORITHM
  8. Connecting to Channels with Delays
  9. REGRET ANALYSIS
  10. EXPERIMENTS
  11. CONCLUSION
  12. LINEAR PROGRAM FORMULATION
  13. Observation
  14. MISSING PROOFS
  15. Proof of Lemma \ref{['lem:schedule-endtime']}
  16. ...and 2 more sections

Key Result

Lemma 1

If the scheduling algorithm outlined in Algorithm scheduling-alg is run for batch $i$, then the end time $T^{(i)}$ of the batch can be bounded as where $\tau = \frac{1}{ \sum\limits_{m=1}^{M} 1 / (\alpha_m / 4^i + 1)}$, $\alpha_m = \lceil 4 \frac{\log{T}}{ \log{(1/\epsilon_m)}}\rceil -1$, $K$ is the number of actions, and $M$ is the number of agents.

Figures (2)

  • Figure 1: Comparison Results For Different Numbers Of Agents. From Left To Right, The Plots Show Cumulative Regret As A Function Of Rounds t For (a) 4 Agents, (b) 20 Agents, and (c) 40 Agents, Respectively.
  • Figure 2: Same Scenario As In Figure \ref{['fig:k10_m4-20-40']} (b) With Worse Channel Quality.

Theorems & Definitions (7)

  • Lemma 1
  • Theorem 1
  • Theorem 2
  • Claim 1
  • Lemma 1
  • Proposition 1
  • Proposition 1