Constant-Memory Strategies in Stochastic Games: Best Responses and Equilibria
Fengming Zhu, Fangzhen Lin
TL;DR
This paper investigates constant-memory strategies in stochastic games, focusing on how bounded-history play shapes best responses and equilibria under bounded memory. It proves that when opponents employ $K$-memory strategies, a $K$-memory best response exists (and can be deterministic), and that for any finite $K$ there is a Nash equilibrium in constant-memory strategies. It also shows that a mixed constant-memory opponent need not be encapsulated by a single constant-memory strategy, and that computing best responses to such mixtures can be computationally hard, reducing to infinite-horizon CMDP/POMDP planning with bidirectional reductions. Theoretical insights are complemented by empirical validation on IPD, ITD, and Pursuit domains, illustrating how memory length impacts equilibria and strategic behavior; the authors also provide code to enable replication and further study.
Abstract
Stochastic games have become a prevalent framework for studying long-term multi-agent interactions, especially in the context of multi-agent reinforcement learning. In this work, we comprehensively investigate the concept of constant-memory strategies in stochastic games. We first establish some results on best responses and Nash equilibria for behavioral constant-memory strategies, followed by a discussion on the computational hardness of best responding to mixed constant-memory strategies. Those theoretic insights are later verified on several sequential decision-making testbeds, including the $\textit{Iterated Prisoner's Dilemma}$, the $\textit{Iterated Traveler's Dilemma}$, and the $\textit{Pursuit}$ domain. This work aims to enhance the understanding of theoretical issues in single-agent planning under multi-agent systems, and uncover the connection between decision models in single-agent and multi-agent contexts. The code is available at $\texttt{https://github.com/Fernadoo/Const-Mem.}$