Beyond Theorems: A Counterexample to Potential Markov Game Criteria
Fatemeh Fardno, Seyed Majid Zahedi
TL;DR
The paper questions whether a relaxed criterion for Markov potential games suffices to guarantee that a deterministic stationary Nash equilibrium can be found by solving a dual MDP. It constructs a continuous-space, infinite-horizon counterexample that satisfies the proposed OPSG and state-transitivity conditions yet yields a Nash equilibrium that differs from the dual-MDP optimum. This finding refutes the claimed equivalence and challenges the practical utility of the relaxed conditions for guiding independent-learning algorithms. The result highlights the need for stronger or alternative assumptions to ensure efficient computation of equilibria in multi-agent stochastic settings.
Abstract
There are only limited classes of multi-player stochastic games in which independent learning is guaranteed to converge to a Nash equilibrium. Markov potential games are a key example of such classes. Prior work has outlined sets of sufficient conditions for a stochastic game to qualify as a Markov potential game. However, these conditions often impose strict limitations on the game's structure and tend to be challenging to verify. To address these limitations, Mguni et al. [12] introduce a relaxed notion of Markov potential games and offer an alternative set of necessary conditions for categorizing stochastic games as potential games. Under these conditions, the authors claim that a deterministic Nash equilibrium can be computed efficiently by solving a dual Markov decision process. In this paper, we offer evidence refuting this claim by presenting a counterexample.