Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Solving Hierarchical Information-Sharing Dec-POMDPs: An Extensive-Form Game Approach

Johan Peralez, Aurélien Delage, Olivier Buffet, Jilles S. Dibangoye

TL;DR

This work tackles scalable decision-making for decentralized partially observable environments by exploiting hierarchical information sharing (HIS). It reframes Dec-POMDPs as single-player occupancy-state problems and further decomposes single-stage subgames into extensive-form games, enabling per-player decisions without sacrificing optimality. The authors introduce nested-occupancy states and associated equivalence relations to compress histories, and they present a PBVI-based algorithm (hPBVI) that leverages these structures to achieve exponential reductions in Backups and improved scalability, demonstrated up to 10 agents. The approach offers practical, provable guarantees and demonstrates competitive performance against local methods, with potential broad impact for high-stakes, structure-rich multi-agent domains. The work thus provides a bridge from extensive-form game theory to scalable global solutions for large teams of agents under hierarchical information flow.

Abstract

A recent theory shows that a multi-player decentralized partially observable Markov decision process can be transformed into an equivalent single-player game, enabling the application of \citeauthor{bellman}'s principle of optimality to solve the single-player game by breaking it down into single-stage subgames. However, this approach entangles the decision variables of all players at each single-stage subgame, resulting in backups with a double-exponential complexity. This paper demonstrates how to disentangle these decision variables while maintaining optimality under hierarchical information sharing, a prominent management style in our society. To achieve this, we apply the principle of optimality to solve any single-stage subgame by breaking it down further into smaller subgames, enabling us to make single-player decisions at a time. Our approach reveals that extensive-form games always exist with solutions to a single-stage subgame, significantly reducing time complexity. Our experimental results show that the algorithms leveraging these findings can scale up to much larger multi-player games without compromising optimality.

Solving Hierarchical Information-Sharing Dec-POMDPs: An Extensive-Form Game Approach

TL;DR

This work tackles scalable decision-making for decentralized partially observable environments by exploiting hierarchical information sharing (HIS). It reframes Dec-POMDPs as single-player occupancy-state problems and further decomposes single-stage subgames into extensive-form games, enabling per-player decisions without sacrificing optimality. The authors introduce nested-occupancy states and associated equivalence relations to compress histories, and they present a PBVI-based algorithm (hPBVI) that leverages these structures to achieve exponential reductions in Backups and improved scalability, demonstrated up to 10 agents. The approach offers practical, provable guarantees and demonstrates competitive performance against local methods, with potential broad impact for high-stakes, structure-rich multi-agent domains. The work thus provides a bridge from extensive-form game theory to scalable global solutions for large teams of agents under hierarchical information flow.

Abstract

A recent theory shows that a multi-player decentralized partially observable Markov decision process can be transformed into an equivalent single-player game, enabling the application of \citeauthor{bellman}'s principle of optimality to solve the single-player game by breaking it down into single-stage subgames. However, this approach entangles the decision variables of all players at each single-stage subgame, resulting in backups with a double-exponential complexity. This paper demonstrates how to disentangle these decision variables while maintaining optimality under hierarchical information sharing, a prominent management style in our society. To achieve this, we apply the principle of optimality to solve any single-stage subgame by breaking it down further into smaller subgames, enabling us to make single-player decisions at a time. Our approach reveals that extensive-form games always exist with solutions to a single-stage subgame, significantly reducing time complexity. Our experimental results show that the algorithms leveraging these findings can scale up to much larger multi-player games without compromising optimality.
Paper Structure (32 sections, 11 theorems, 57 equations, 16 figures, 1 table, 1 algorithm)

This paper contains 32 sections, 11 theorems, 57 equations, 16 figures, 1 table, 1 algorithm.

Table of Contents

  1. Background
  2. Notations.
  3. Multi-Player Formulation
  4. Limitations of Multi-Player Formulations
  5. Single-Player Reformulation
  6. Limitations of Single-Player Reformulations
  7. Hierarchical Information Sharing
  8. From Single-Stage to Extensive-Form Games
  9. Optimally Solving $G_{s_\tau}^{\beta_\tau}$ As $\bar{G}_{s_\tau}^{\beta_\tau}$
  10. Nested-Occupancy States
  11. Near-Optimally Solving $M'$ Under HIS
  12. Experiments
  13. Discussion
  14. The PBVI Algorithm
  15. Proof of Theorem \ref{['thm:sufficiency']}
  16. ...and 17 more sections

Key Result

Lemma 1.1

For every game stage $\tau$, the optimal value function $Q^*_\tau\colon S \times A \to \mathbb{R}$ is piecewise-linear and convex over occupancy states and joint decision rules. Alternatively, there exists a finite collection ${\mathcal{Q}}_\tau \subseteq \{ \beta^{a_{\tau+1:}}_\tau| a_{\tau+1:}\in with boundary condition $\alpha^{\cdot}_{\ell}(\cdot) =\beta^{\cdot}_{\ell}(\cdot) \doteq 0$.

Figures (16)

  • Figure 1: V2X information transmitted to vehicles in the platooning.
  • Figure 2: The search space for a single-stage subgame from a centralized planner acting sequentially one player at a time, illustrated as an AND/OR tree. OR nodes (triangle) represent alternative ways to solve $\bar{G}_{s_\tau}^{\beta_\tau}$. AND nodes (circle) represent subproblem alternatives to be solved. Best viewed in color.
  • Figure 3: Average backup time for the recycling problem with different numbers of agents.
  • Figure 4: Anytime values for the recycling problem with teams of size $n\in\{3,4,6,8\}$ and planning horizon $\ell = 30$.
  • Figure 5: Average Backup Time for the tiger problem and different numbers of players.
  • ...and 11 more figures

Theorems & Definitions (22)

  • Lemma 1.1
  • Definition 2.1
  • Theorem 2.2
  • proof
  • Theorem 2.3: Proof in Appendix \ref{['appendix:thm:sufficiency']}
  • Theorem 2.4: Proof in Appendix \ref{['appendix:thm:compression']}
  • Theorem 3.1: Proof in Appendix \ref{['appendix:thm:error:bound']}
  • Theorem 3.2
  • proof
  • proof
  • ...and 12 more