Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Simulation-Based Optimistic Policy Iteration For Multi-Agent MDPs with Kullback-Leibler Control Cost

Khaled Nakhleh, Ceyhun Eksin, Sabit Ekin

TL;DR

This paper uses the separable structure of the instantaneous cost to show that the policy improvement step follows a Boltzmann distribution that depends on the current value function estimate and the uncontrolled transition probabilities and allows agents to compute the improved joint policy independently.

Abstract

This paper proposes an agent-based optimistic policy iteration (OPI) scheme for learning stationary optimal stochastic policies in multi-agent Markov Decision Processes (MDPs), in which agents incur a Kullback-Leibler (KL) divergence cost for their control efforts and an additional cost for the joint state. The proposed scheme consists of a greedy policy improvement step followed by an m-step temporal difference (TD) policy evaluation step. We use the separable structure of the instantaneous cost to show that the policy improvement step follows a Boltzmann distribution that depends on the current value function estimate and the uncontrolled transition probabilities. This allows agents to compute the improved joint policy independently. We show that both the synchronous (entire state space evaluation) and asynchronous (a uniformly sampled set of substates) versions of the OPI scheme with finite policy evaluation rollout converge to the optimal value function and an optimal joint policy asymptotically. Simulation results on a multi-agent MDP with KL control cost variant of the Stag-Hare game validates our scheme's performance in terms of minimizing the cost return.

Simulation-Based Optimistic Policy Iteration For Multi-Agent MDPs with Kullback-Leibler Control Cost

TL;DR

This paper uses the separable structure of the instantaneous cost to show that the policy improvement step follows a Boltzmann distribution that depends on the current value function estimate and the uncontrolled transition probabilities and allows agents to compute the improved joint policy independently.

Abstract

This paper proposes an agent-based optimistic policy iteration (OPI) scheme for learning stationary optimal stochastic policies in multi-agent Markov Decision Processes (MDPs), in which agents incur a Kullback-Leibler (KL) divergence cost for their control efforts and an additional cost for the joint state. The proposed scheme consists of a greedy policy improvement step followed by an m-step temporal difference (TD) policy evaluation step. We use the separable structure of the instantaneous cost to show that the policy improvement step follows a Boltzmann distribution that depends on the current value function estimate and the uncontrolled transition probabilities. This allows agents to compute the improved joint policy independently. We show that both the synchronous (entire state space evaluation) and asynchronous (a uniformly sampled set of substates) versions of the OPI scheme with finite policy evaluation rollout converge to the optimal value function and an optimal joint policy asymptotically. Simulation results on a multi-agent MDP with KL control cost variant of the Stag-Hare game validates our scheme's performance in terms of minimizing the cost return.

Paper Structure

This paper contains 16 sections, 5 theorems, 35 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Related Work and Contributions
  3. Simulation-Based Optimistic Policy Iteration
  4. Decentralized Learning in Multi-Agent MDPs and Markov Games
  5. Multi-Agent MDPs With KL Control Cost
  6. Stochastic Joint and Marginal Policies
  7. Decomposition of the One-Step Cost Function
  8. The KL Evaluation and KL Optimal Bellman Operators
  9. Multi-Agent Simulation-Based Kullback-Leibler Controlled Optimistic Policy Iteration $\texttt{KLC-OPI}$
  10. The Scheme Description
  11. Asymptotic Convergence
  12. The Asynchronous $\texttt{KLC-OPI}$ Scheme
  13. Simulations
  14. A Multi-Agent MDP: Stag-Hare
  15. $\texttt{ASYNC-KLC-OPI}$ Simulation Results
  16. ...and 1 more sections

Key Result

Lemma 4.1

At iteration $k$ and for any $i \in \mathcal{N}$, the joint policy $\pi^{(i)}_{P_0,k+1}$ given $P_0$ that minimizes the discounted return of the policy evaluation step in scheme:kl-opi follows a Boltzmann distribution where $Z_{i,k}$ for agent $i$ is the Cole-Hopf transformation of the value function such that $Z_{i,k}(s) = e^{-V_{i,k}(s)}$ state-wise in iteration $k$.

Figures (2)

  • Figure 1: A multi-agent MDP with KL control cost: Two hunters hunting either hares or a stag on a $5 \times 5$ gridworld. See Section \ref{['section:simulations']} for details.
  • Figure 2: $\texttt{ASYNC-KLC-OPI}$ performance on the multi-agent MDP with KL control cost variant of the Stag-Hare game averaged over $10$ simulation runs.

Theorems & Definitions (11)

  • Lemma 4.1
  • proof
  • Remark 4.2
  • Remark 4.3
  • Theorem 4.4
  • Lemma 4.5: Lemma 2.c in tsitsiklis2002
  • Proposition 4.6
  • proof
  • proof : Proof (Theorem \ref{['theorem:global-error-convergence']})
  • Corollary 5.2
  • ...and 1 more