Asymptotically Optimal Policies for Weakly Coupled Markov Decision Processes

TL;DR

This work addresses maximizing long-run average reward for collections of weakly coupled Markov decision processes with multiple actions and inequality constraints. It develops a mean-field fluid-relaxation framework that reduces the problem to solving a deterministic occupancy-measure control problem and then constructing discrete policies that are asymptotically optimal as the number of processes grows. The authors prove the mean-field limit, provide sufficient conditions for the existence of fluid solutions, and give explicit constructions in important cases such as multi-armed restless bandits and inequality-constrained problems, along with numerical demonstrations. The approach yields policies with strong theoretical guarantees and practical rounding schemes, offering robust alternatives to traditional restless-bandit policies in infinite-horizon settings with general constraints.

Abstract

We consider the problem of maximizing the expected average reward obtained over an infinite time horizon by $n$ weakly coupled Markov decision processes. Our setup is a substantial generalization of the multi-armed restless bandit problem that allows for multiple actions and constraints. We establish a connection with a deterministic and continuous-variable control problem where the objective is to maximize the average reward derived from an occupancy measure that represents the empirical distribution of the processes when $n \to \infty$. We show that a solution of this fluid problem can be used to construct policies for the weakly coupled processes that achieve the maximum expected average reward as $n \to \infty$, and we give sufficient conditions for the existence of solutions. Under certain assumptions on the constraints, we prove that these conditions are automatically satisfied if the unconstrained single-process problem admits a suitable unichain and aperiodic policy. In particular, the assumptions include multi-armed restless bandits and a broad class of problems with multiple actions and inequality constraints. Also, the policies can be constructed in an explicit way in these cases. Our theoretical results are complemented by several concrete examples and numerical experiments, which include multichain setups that are covered by the theoretical results.

Figures (4)

• Figure 1: The left panel shows the reward obtained per process and per time step as a function of the state of the process and the action selected. The right panel charts the optimal value of the fluid relaxation, and the gain of the discrete control $\phi_n$ when all the batteries are initially empty.
• Figure 2: The left panel and right panels correspond to the nonindexable problem and the problem where the global attractivity property does not hold for the Whittle index and LP-priority policies, respectively.
• Figure 3: The diagrams depict the transition probabilities for both actions. The plot compares the optimal value of the fluid relaxation and the gain of several policies. The gains were computed by simulating systems where all the arms are initially in state $0$.
• Figure 4: The diagrams depict the transition probabilities and rewards for both actions; all the transitions are deterministic. The chart compares the optimal value of the fluid relaxation and the gain of different policies; the gains were computed by simulating systems where all the arms are initially in state $0$.

Theorems & Definitions (31)

• Definition 1
• Proposition 1
• Definition 2
• Definition 3
• Lemma 1
• Proposition 2
• proof
• Lemma 2
• proof
• Theorem 1