Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Bandit Whisperer: Communication Learning for Restless Bandits

Yunfan Zhao, Tonghan Wang, Dheeraj Nagaraj, Aparna Taneja, Milind Tambe

TL;DR

This work addresses RMABs under systematic reward noise by introducing a learning framework where arms communicate via sharing Q-network parameters. Modeling the setup as a Multi-Agent MDP with binary communication actions, the method uses a decomposed Q-network to handle the combinatorial joint action space and a joint communication reward based on RMAB performance gains. Theoretical results identify conditions under which communication reduces sample complexity, and experiments on Synthetic, SIS, and ARMMAN data demonstrate robust improvements over baselines and adaptive communication strategies. The approach offers a principled, scalable way to mitigate data-errors in resource-constrained, dynamic systems with real-world relevance.

Abstract

Applying Reinforcement Learning (RL) to Restless Multi-Arm Bandits (RMABs) offers a promising avenue for addressing allocation problems with resource constraints and temporal dynamics. However, classic RMAB models largely overlook the challenges of (systematic) data errors - a common occurrence in real-world scenarios due to factors like varying data collection protocols and intentional noise for differential privacy. We demonstrate that conventional RL algorithms used to train RMABs can struggle to perform well in such settings. To solve this problem, we propose the first communication learning approach in RMABs, where we study which arms, when involved in communication, are most effective in mitigating the influence of such systematic data errors. In our setup, the arms receive Q-function parameters from similar arms as messages to guide behavioral policies, steering Q-function updates. We learn communication strategies by considering the joint utility of messages across all pairs of arms and using a Q-network architecture that decomposes the joint utility. Both theoretical and empirical evidence validate the effectiveness of our method in significantly improving RMAB performance across diverse problems.

The Bandit Whisperer: Communication Learning for Restless Bandits

TL;DR

This work addresses RMABs under systematic reward noise by introducing a learning framework where arms communicate via sharing Q-network parameters. Modeling the setup as a Multi-Agent MDP with binary communication actions, the method uses a decomposed Q-network to handle the combinatorial joint action space and a joint communication reward based on RMAB performance gains. Theoretical results identify conditions under which communication reduces sample complexity, and experiments on Synthetic, SIS, and ARMMAN data demonstrate robust improvements over baselines and adaptive communication strategies. The approach offers a principled, scalable way to mitigate data-errors in resource-constrained, dynamic systems with real-world relevance.

Abstract

Applying Reinforcement Learning (RL) to Restless Multi-Arm Bandits (RMABs) offers a promising avenue for addressing allocation problems with resource constraints and temporal dynamics. However, classic RMAB models largely overlook the challenges of (systematic) data errors - a common occurrence in real-world scenarios due to factors like varying data collection protocols and intentional noise for differential privacy. We demonstrate that conventional RL algorithms used to train RMABs can struggle to perform well in such settings. To solve this problem, we propose the first communication learning approach in RMABs, where we study which arms, when involved in communication, are most effective in mitigating the influence of such systematic data errors. In our setup, the arms receive Q-function parameters from similar arms as messages to guide behavioral policies, steering Q-function updates. We learn communication strategies by considering the joint utility of messages across all pairs of arms and using a Q-network architecture that decomposes the joint utility. Both theoretical and empirical evidence validate the effectiveness of our method in significantly improving RMAB performance across diverse problems.
Paper Structure (21 sections, 8 theorems, 34 equations, 4 figures, 5 tables)

This paper contains 21 sections, 8 theorems, 34 equations, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. RMABs with Noisy Arms
  4. Reinforcement Learning in RMABs
  5. RMAB Learning with Communication
  6. Define the Communication Learning Problem
  7. Communication Actions and Transition Dynamics
  8. Communication Reward
  9. Decomposed Communication Q Function
  10. Theoretical Analysis
  11. Experiments
  12. Closing Remarks
  13. Ethics and Real-world ARMMAN Dataset
  14. Secondary Analysis
  15. Consent and Data Usage
  16. ...and 6 more sections

Key Result

Proposition 1

For the Q-learning of arm $i$, when with a probability at least $(1-\delta)^2$ for any $0<\delta<1$, compared to learning without communication, choosing to receive communication and using behavior policy $\pi_{\nu_i}$ leads to better sample complexity in the worst case for learning $\epsilon_e$-optimal Q function such that $\|Q_i(s,a

Figures (4)

  • Figure 1: Performance (interquartile mean agarwal2021deep and standard error of return over 200 random seeds) of our method, baselines, and ablations in three environments with different numbers of arms $N$ and resource budgets $B$. Communication learning starts at epoch 200. We present results on two additional baseline in Tables \ref{['table:noise_level_ablation']} and \ref{['table:num_noisy_arms_ablation']} (see Appendix B for more ablations).
  • Figure 2: The change in the proportion of noise-free senders and receiver within all activated channels during the learning process. Results are for Synthetic (N=21, B=15), SIS (N=32, B=16), and ARMMAN (N=48, B=16).
  • Figure 3: The example MDP family where communication from a noise-free arm to an noisy arm is helpful.
  • Figure 4: The example RMAB family where communication from a noise-free arm to an noisy arm is useless.

Theorems & Definitions (13)

  • Definition 1: The Communication MDP
  • Proposition 1: Useful Communication
  • Proposition 2: Sparse Communication
  • Lemma 2
  • Lemma 2
  • Remark 3
  • Proposition 3: Sparse Communication
  • proof
  • Proposition 3: Useful Communication
  • Lemma 3
  • ...and 3 more