Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability

Eline M. Bovy, Caleb Probine, Marnix Suilen, Ufuk Topcu, Nils Jansen

TL;DR

This work tackles planning under discrete model uncertainty in partially observable domains by introducing adversarial-belief POMDPs (AB-POMDPs) and showing they subsume multi-environment POMDPs (ME-POMDPs) through reductions to one-sided POSGs. It formalizes the theoretical relationships between AB-POMDPs, ME-POMDPs, MO-POMDPs, and PO-MEMDPs, and develops exact (LP-based) and approximate (AB-HSVI) algorithms for robust planning. Experimental results on Bird- and RockSample-inspired benchmarks demonstrate the feasibility of computing robust policies while highlighting trade-offs in convergence time and formulation (AB-POMDP vs ME-POMDP). Overall, the paper provides a principled framework and practical algorithms for robust decision-making under discrete model uncertainty in partially observable settings.

Abstract

Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, i.e., a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs, which we call adversarial-belief POMDPs (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.

Multi-Environment POMDPs: Discrete Model Uncertainty Under Partial Observability

TL;DR

This work tackles planning under discrete model uncertainty in partially observable domains by introducing adversarial-belief POMDPs (AB-POMDPs) and showing they subsume multi-environment POMDPs (ME-POMDPs) through reductions to one-sided POSGs. It formalizes the theoretical relationships between AB-POMDPs, ME-POMDPs, MO-POMDPs, and PO-MEMDPs, and develops exact (LP-based) and approximate (AB-HSVI) algorithms for robust planning. Experimental results on Bird- and RockSample-inspired benchmarks demonstrate the feasibility of computing robust policies while highlighting trade-offs in convergence time and formulation (AB-POMDP vs ME-POMDP). Overall, the paper provides a principled framework and practical algorithms for robust decision-making under discrete model uncertainty in partially observable settings.

Abstract

Multi-environment POMDPs (ME-POMDPs) extend standard POMDPs with discrete model uncertainty. ME-POMDPs represent a finite set of POMDPs that share the same state, action, and observation spaces, but may arbitrarily vary in their transition, observation, and reward models. Such models arise, for instance, when multiple domain experts disagree on how to model a problem. The goal is to find a single policy that is robust against any choice of POMDP within the set, i.e., a policy that maximizes the worst-case reward across all POMDPs. We generalize and expand on existing work in the following way. First, we show that ME-POMDPs can be generalized to POMDPs with sets of initial beliefs, which we call adversarial-belief POMDPs (AB-POMDPs). Second, we show that any arbitrary ME-POMDP can be reduced to a ME-POMDP that only varies in its transition and reward functions or only in its observation and reward functions, while preserving (optimal) policies. We then devise exact and approximate (point-based) algorithms to compute robust policies for AB-POMDPs, and thus ME-POMDPs. We demonstrate that we can compute policies for standard POMDP benchmarks extended to the multi-environment setting.

Paper Structure

This paper contains 27 sections, 9 theorems, 75 equations, 11 figures, 7 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Preliminaries
  5. Solving POMDPs
  6. One-sided Partially Observable Stochastic Games
  7. Adversarial-Belief and Multi-Environment POMDPs
  8. Adversarial-Belief POMDPs
  9. Multi-Environment POMDPs
  10. Restricted Models and Reductions
  11. Algorithms for AB-POMDPs
  12. Computing Policies by Solving Linear Programs
  13. Adversarial-Belief HSVI
  14. Experimental Evaluation
  15. Conclusion
  16. ...and 12 more sections

Key Result

Theorem 1

Let $\mathsf{M} = (S,A,Z,T,O,R,\Delta(Q) ,\gamma, H)$ be an AB-POMDP. We define the associated one-sided POSG $\mathcal{G} = ((S\times\{1,2\})\cup \{\perp\}, A, Q,Z \cup \{\top\}, \hat{T}, \hat{O}, \hat{R}, \delta_\perp, \gamma,H+1)$ where and $\hat{R}(\hat{s},a,q) = 0$ if $\hat{s} = \bot$ and $\hat{R}(\hat{s},a,q) = R(s,a)/\gamma$ when $\hat{s} = (s,j)$, for all $\hat{s} \in (S\times \{1,2\}) \c

Figures (11)

  • Figure 1: ME-POMDPs are a rich class of models between POMDPs and one-sided POSGs. Arrows from class A to class B indicate that we can transform models in class A to models in class B. The transformed model size is polynomial in the original model size for all arrows not marked by $*$. Unmarked arrows are trivial reductions. We define MO-POMDPs and PO-MEMDPs in Section \ref{['sec:rest_mod']}.
  • Figure 1: Lower bound value, time of convergence, and left-over gap between upper and lower bound of the Bird problem for various problem sizes and model types.
  • Figure 2: Lower bound value, time of convergence, and left-over gap between upper and lower bound of the RockSample problem for various problem sizes with rocks nearby or far away.
  • Figure 3: Convergence time of RockSample problems modeled as AB-POMDPs vs. ME-POMDPs.
  • Figure 4: Lower bound value, convergence time, and gap for various RockSample problems.
  • ...and 6 more figures

Theorems & Definitions (14)

  • Definition 1: POMDP
  • Definition 2: POSG
  • Definition 3: AB-POMDP
  • Theorem 1
  • Definition 4: ME-POMDP
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Theorem 5
  • Remark 1
  • ...and 4 more