Model-Free Learning and Optimal Policy Design in Multi-Agent MDPs Under Probabilistic Agent Dropout

TL;DR

It is shown that the robust MDP value can be estimated with samples generated by the predropout system, meaning that robust policies can be found before dropout occurs, and a policy importance sampling routine is proposed that performs policy evaluation for dropout scenarios while controlling the existing system with good predropout policies.

Abstract

This work studies a multi-agent Markov decision process (MDP) that can undergo agent dropout and the computation of policies for the post-dropout system based on control and sampling of the pre-dropout system. The central planner's objective is to find an optimal policy that maximizes the value of the expected system given a priori knowledge of the agents' dropout probabilities. For MDPs with a certain transition independence and reward separability structure, we assume that removing agents from the system forms a new MDP comprised of the remaining agents with new state and action spaces, transition dynamics that marginalize the removed agents, and rewards that are independent of the removed agents. We first show that under these assumptions, the value of the expected post-dropout system can be represented by a single MDP; this "robust MDP" eliminates the need to evaluate all $2^N$ realizations of the system, where N denotes the number of agents. More significantly, in a model-free context, it is shown that the robust MDP value can be estimated with samples generated by the pre-dropout system, meaning that robust policies can be found before dropout occurs. This fact is used to propose a policy importance sampling (IS) routine that performs policy evaluation for dropout scenarios while controlling the existing system with good pre-dropout policies. The policy IS routine produces value estimates for both the robust MDP and specific post-dropout system realizations and is justified with exponential confidence bounds. Finally, the utility of this approach is verified in simulation, showing how structural properties of agent dropout can help a controller find good post-dropout policies before dropout occurs.

Figures (5)

• Figure 1: Example system graph. Part (a) shows system under no dropout, i.e., the nominal graph. Part (b) shows a system dropout realization. Part (c) shows the expected system, where the weight of each link is scaled by the corresponding $\beta$.
• Figure 2: Experimental comparison of various policies on the pre- and post-dropout system. $N=4$, $|\mathcal{X}_n| = 2$, $|\mathcal{A}_n|=2$. Note that the robust policy performs almost as well as the optimal pre-dropout policy for $t<500$, and is much better than the optimal pre-dropout policy for $t>500$. In addition, the central planner can pre-compute the robust policy with policy IS.
• Figure 3: Experimental demonstration of the optimality gap and associated upper bound (Lemma \ref{['opt gap']}). $N=4$, $|\mathcal{X}_n| = 3$, $|\mathcal{A}_n| = 3$, and the rewards were assigned such that $|r_n(x_n,\alpha_n|w_n=1)| \leq 1$ and $r_n(x_n,\alpha_n|w_n=0) = 0$.
• Figure 4: Empirical losses produced by the robust policy and optimal pre-dropout policy on post-dropout system. $N=5$, $|\mathcal{X}_n| = 2$, $|\mathcal{A}_n|=2$, averaged across all dropout combinations, and 1000 experiments. Error bars show max/min loss over the dropout combinations. The robust policy performs better on average than the pre-dropout policy when for less than half of the agents, and it performs better in the maximum for $\beta\in\{0.2, 0.4, 0.6\}$.
• Figure 5: Performance of policy IS to evaluate a candidate policy. $N=5$, $|\mathcal{X}_n|=2$, $|\mathcal{A}_n|=2$, $|D|=100$, $H=500$, and $H_{\mu} = 5000$. Note that the policy IS estimator approximates the true robust value. The candidate policy value of the robust MDP is close to the pre-dropout MDP. This particular candidate policy on the robust system achieves $\sim$60% of the optimal pre-dropout MDP value.

Theorems & Definitions (24)

• Definition 1
• Definition 2
• Theorem 1
• proof
• Theorem 2
• proof
• Remark 1
• Theorem 3
• proof
• Lemma 1