Approximating the Uniform Value in Hidden Stochastic Games with Doeblin Conditions

TL;DR

The existence of the uniform value is proved and an algorithm to approximate it is provided and it is shown that, in the hidden setting, ergodicity does not guarantee the Doeblin condition.

Abstract

In \emph{zero-sum two-player hidden stochastic games}, players observe partial information about the state. We address: $(i)$ the existence of the \emph{uniform value}, i.e., a limiting average payoff that both players can guarantee for sufficiently long durations, and $(ii)$ the existence of an algorithm to approximate it. Previous work shows that, in the general case, the uniform value may fail to exist, and, even when it does, there need not exist an algorithm to compute or approximate it. Therefore, we consider the \emph{Doeblin condition} in hidden stochastic games, requiring that, after a sufficiently long time, the posterior beliefs have a uniformly positive probability of resetting to one of finitely many neighborhoods in the belief space. We prove the existence of the uniform value and provide an algorithm to approximate it. We identify sufficient conditions, namely \emph{ergodicity} in the blind setting (when the signal is uninformative) and \emph{primitivity} in the hidden setting (when there are multiple signals). Moreover, we show that, in the hidden setting, ergodicity does not guarantee the Doeblin condition. Our results are new even for the one-player setting, i.e., partially observable Markov decision processes.

Figures (1)

• Figure 1: State transition diagram of $\Gamma$. The triangle facing up are states controlled by Player $1$ while the triangle facing down are states controlled by Player $2$.

Theorems & Definitions (22)

• Definition 2.1: Decision version of computing the uniform value
• Definition 2.2: Decision version of approximating the uniform value
• Definition 3.1: Doeblin hidden stochastic game
• Theorem 3.2
• Corollary 3.3
• Definition 3.4: Ergodic blind stochastic game
• Theorem 3.5
• Definition 3.6: Primitive hidden stochastic game
• Theorem 3.7
• Lemma 4.1