Learning Recommender Mechanisms for Bayesian Stochastic Games

TL;DR

The paper tackles coordinating among self-interested agents with private information in Bayesian stochastic games by learning a recommender mechanism that maps reported types to deterministic Markov stationary policies. It introduces ReMBo, a bi-level reinforcement learning framework where the outer loop optimizes a parametric mechanism $\mathcal{M}$ while inner loops solve incentive-compatibility and individual-rationality deviations via differentiable surrogates, including $Q$- and $V$-functions and straight-through $\text{Gumbel-Softmax}$ for discrete actions. The objective blends social welfare with incentive penalties, using a Lagrangian relaxation with weights $\lambda_1$, $\lambda_2$ (equivalently, $\alpha_0$, $\alpha_1$, $\alpha_2$) to balance objectives. Experiments across repeated bimatrix games, lane-changing, and congestion scenarios show that ReMBo achieves social welfare competitive with cooperative MARL baselines while delivering significantly improved incentive properties, aided by a shared replay buffer that enhances exploration. This work enables scalable, incentive-aligned recommendations in complex dynamic multi-agent environments without monetary transfers.

Abstract

An important challenge in non-cooperative game theory is coordinating on a single (approximate) equilibrium from many possibilities - a challenge that becomes even more complex when players hold private information. Recommender mechanisms tackle this problem by recommending strategies to players based on their reported type profiles. A key consideration in such mechanisms is to ensure that players are incentivized to participate, report their private information truthfully, and follow the recommendations. While previous work has focused on designing recommender mechanisms for one-shot and extensive-form games, these approaches cannot be effectively applied to stochastic games, particularly if we constrain recommendations to be Markov stationary policies. To bridge this gap, we introduce a novel bi-level reinforcement learning approach for automatically designing recommender mechanisms in Bayesian stochastic games. Our method produces a mechanism represented by a parametric function (such as a neural network), and is therefore highly efficient at execution time. Experimental results on two repeated and two stochastic games demonstrate that our approach achieves social welfare levels competitive with cooperative multi-agent reinforcement learning baselines, while also providing significantly improved incentive properties.

Figures (13)

• Figure 1: Results for Chicken (top) and Stag Hunt (bottom). Left: social welfare, middle: IC deviation, right: IR deviation.
• Figure 2: Results for the lane-changing game. Top: 15 agents. Bottom: 30 agents. Left: social welfare, middle: IC deviation, and right: IR deviation.
• Figure 3: Results for congestion games. Top: 3-destination games. Bottom: intersection games. Left: social welfare, middle: IC deviation, and right: IR deviation.
• Figure 4: (a) MLP actor architecture for the mechanism $\mathcal{M}_{\phi^n}^n(\theta)$, (b) MLP actor architecture for the mechanism $\mathcal{M}_{\phi^{n-1}}^{n-1}(\theta_{-i})$, (c) MLP actor architecture for the agent deviation policy $\pi'_{\phi^{n-1}, i}(\theta_i)$
• Figure 5: (a) CNN actor architecture for the mechanism $\mathcal{M}_{\phi^n}^n(\theta)$, (b) CNN actor architecture for the mechanism $\mathcal{M}_{\phi^{n-1}}^{n-1}(\theta_{-i})$, (c) CNN actor architecture for the agent deviation policy $\pi'_{\phi^{n-1}, i}(\theta_i)$

Theorems & Definitions (3)

• Definition 1
• Definition 2
• Theorem 1