On characterization and existence of constrained correlated equilibria in Markov games
Tingting Ni, Anna Maddux, Maryam Kamgarpour
TL;DR
This work analyzes constrained correlated equilibria in finite-horizon Markov games with coupling constraints. It proves that the most general equilibrium notion is captured by either Markovian stochastic modifications or by convex combinations of Markovian deterministic modifications, enabling practical learning and verification. Existence results show that strong Slater’s condition is necessary under playerwise coupling but can be weakened under common coupling constraints, with a Kakutani fixed-point construction and a supporting LP framework driving the argument. The findings illuminate when constrained CE exists and how to compute or learn them, with implications for environmental, energy, and transportation applications where joint constraints are pervasive.
Abstract
Markov games with coupling constraints provide a natural framework to study constrained decision-making involving self-interested agents, where the feasibility of an individual agent's strategy depends on the joint strategies of the others. Such games arise in numerous real-world applications involving safety requirements and budget caps, for example, in environmental management, electricity markets, and transportation systems. While correlated equilibria have emerged as an important solution concept in unconstrained settings due to their computational tractability and amenability to learning, their constrained counterparts remain less explored. In this paper, we study constrained correlated equilibria-feasible policies where any unilateral modifications are either unprofitable or infeasible. We first characterize the constrained correlated equilibrium showing that different sets of modifications result in an equivalent notion, a result which may enable efficient learning algorithms. We then address existence conditions. In particular, we show that a strong Slater-type condition is necessary in games with playerwise coupling constraints, but can be significantly weakened when all players share common coupling constraints. Under this relaxed condition, we prove the existence of a constrained correlated equilibrium.