Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Actor-Dual-Critic Dynamics for Zero-sum and Identical-Interest Stochastic Games

Ahmed Said Donmez, Yuksel Arslantas, Muhammed O. Sayin

TL;DR

This work introduces an independent, payoff-based learning framework called Actor-Dual-Critic (ADC) for stochastic games, combining a fast, payoff-responsive critic with a slow, value-focused critic to address non-stationarity in multi-agent environments. By embedding exploration into an effective stochastic game and employing a three-timescale learning scheme, the authors prove convergence to approximate Nash equilibria in both two-agent zero-sum and multi-agent identical-interest settings, even under minimal information. Theoretical guarantees are complemented by empirical results showing robust performance and convergence of the Nash gap below a predefined threshold across both classes of games. The approach advances decentralized MARL by providing gradient-free, provably convergent learning dynamics with practical relevance for zero-sum and potential-based multi-agent systems.

Abstract

We propose a novel independent and payoff-based learning framework for stochastic games that is model-free, game-agnostic, and gradient-free. The learning dynamics follow a best-response-type actor-critic architecture, where agents update their strategies (actors) using feedback from two distinct critics: a fast critic that intuitively responds to observed payoffs under limited information, and a slow critic that deliberatively approximates the solution to the underlying dynamic programming problem. Crucially, the learning process relies on non-equilibrium adaptation through smoothed best responses to observed payoffs. We establish convergence to (approximate) equilibria in two-agent zero-sum and multi-agent identical-interest stochastic games over an infinite horizon. This provides one of the first payoff-based and fully decentralized learning algorithms with theoretical guarantees in both settings. Empirical results further validate the robustness and effectiveness of the proposed approach across both classes of games.

Actor-Dual-Critic Dynamics for Zero-sum and Identical-Interest Stochastic Games

TL;DR

This work introduces an independent, payoff-based learning framework called Actor-Dual-Critic (ADC) for stochastic games, combining a fast, payoff-responsive critic with a slow, value-focused critic to address non-stationarity in multi-agent environments. By embedding exploration into an effective stochastic game and employing a three-timescale learning scheme, the authors prove convergence to approximate Nash equilibria in both two-agent zero-sum and multi-agent identical-interest settings, even under minimal information. Theoretical guarantees are complemented by empirical results showing robust performance and convergence of the Nash gap below a predefined threshold across both classes of games. The approach advances decentralized MARL by providing gradient-free, provably convergent learning dynamics with practical relevance for zero-sum and potential-based multi-agent systems.

Abstract

We propose a novel independent and payoff-based learning framework for stochastic games that is model-free, game-agnostic, and gradient-free. The learning dynamics follow a best-response-type actor-critic architecture, where agents update their strategies (actors) using feedback from two distinct critics: a fast critic that intuitively responds to observed payoffs under limited information, and a slow critic that deliberatively approximates the solution to the underlying dynamic programming problem. Crucially, the learning process relies on non-equilibrium adaptation through smoothed best responses to observed payoffs. We establish convergence to (approximate) equilibria in two-agent zero-sum and multi-agent identical-interest stochastic games over an infinite horizon. This provides one of the first payoff-based and fully decentralized learning algorithms with theoretical guarantees in both settings. Empirical results further validate the robustness and effectiveness of the proposed approach across both classes of games.
Paper Structure (17 sections, 79 equations, 1 figure, 1 algorithm)

This paper contains 17 sections, 79 equations, 1 figure, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Stochastic Games
  5. Actor-Dual-Critic Learning Dynamics
  6. Main Results
  7. Effective Stochastic Game Formulation
  8. Three-timescale Dynamics
  9. Convergence Results
  10. Convergence Analysis
  11. Proof for Two-agent Zero-sum Stochastic Games
  12. Proof for Multi-agent Identical-interest Stochastic Games
  13. Rate Characterization
  14. Lower Bound Formulation
  15. Limit Characterization
  16. ...and 2 more sections

Figures (1)

  • Figure 1: The Nash-gap decay for our ADC, the Dec-Q, and the IndDec dynamics. The dashed line is the $\varepsilon$-threshold \ref{['eq:threshold']}. The Nash gap for \ref{['alg:ADC']} decays effectively below this threshold for both zero-sum and identical-interest stochastic games.