Provably Efficient Reinforcement Learning for Adversarial Restless Multi-Armed Bandits with Unknown Transitions and Bandit Feedback
Guojun Xiong, Jian Li
TL;DR
This work tackles adversarial RMAB with unknown transition dynamics and bandit feedback, introducing UCMD-ARMAB to learn under an instantaneous activation constraint. It combines confidence-set based transition estimation, a time-dependent occupancy-measure formulation solved by Online Mirror Descent, a biased adversarial reward estimator for bandit feedback, and a low-complexity Reward Maximizing Index policy to enforce feasibility. The authors prove a regret bound of $\tilde{\mathcal{O}}(H\sqrt{T})$ (up to problem constants) and demonstrate empirical gains over a stochastic baseline in realistic RMAB-like tasks, highlighting practical viability in non-stationary, adversarial environments. The results advance the theoretical and practical understanding of learning in adversarial RMAB with unknown transitions and limited feedback, enabling provably efficient decision-making under tight instantaneous activation constraints.
Abstract
Restless multi-armed bandits (RMAB) play a central role in modeling sequential decision making problems under an instantaneous activation constraint that at most B arms can be activated at any decision epoch. Each restless arm is endowed with a state that evolves independently according to a Markov decision process regardless of being activated or not. In this paper, we consider the task of learning in episodic RMAB with unknown transition functions and adversarial rewards, which can change arbitrarily across episodes. Further, we consider a challenging but natural bandit feedback setting that only adversarial rewards of activated arms are revealed to the decision maker (DM). The goal of the DM is to maximize its total adversarial rewards during the learning process while the instantaneous activation constraint must be satisfied in each decision epoch. We develop a novel reinforcement learning algorithm with two key contributors: a novel biased adversarial reward estimator to deal with bandit feedback and unknown transitions, and a low-complexity index policy to satisfy the instantaneous activation constraint. We show $\tilde{\mathcal{O}}(H\sqrt{T})$ regret bound for our algorithm, where $T$ is the number of episodes and $H$ is the episode length. To our best knowledge, this is the first algorithm to ensure $\tilde{\mathcal{O}}(\sqrt{T})$ regret for adversarial RMAB in our considered challenging settings.