Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Easy as ABCs: Unifying Boltzmann Q-Learning and Counterfactual Regret Minimization

Luca D'Amico-Wong, Hugh Zhang, Marc Lanctot, David C. Parkes

TL;DR

ABCs (Adaptive Branching through Child stationarity), a best-of-both-worlds algorithm combining Boltzmann Q-learning (BQL), a classic reinforcement learning algorithm for single-agent domains, and counterfactual regret minimization (CFR), a central algorithm for learning in multi-agent domains, demonstrates strong performance.

Abstract

We propose ABCs (Adaptive Branching through Child stationarity), a best-of-both-worlds algorithm combining Boltzmann Q-learning (BQL), a classic reinforcement learning algorithm for single-agent domains, and counterfactual regret minimization (CFR), a central algorithm for learning in multi-agent domains. ABCs adaptively chooses what fraction of the environment to explore each iteration by measuring the stationarity of the environment's reward and transition dynamics. In Markov decision processes, ABCs converges to the optimal policy with at most an O(A) factor slowdown compared to BQL, where A is the number of actions in the environment. In two-player zero-sum games, ABCs is guaranteed to converge to a Nash equilibrium (assuming access to a perfect oracle for detecting stationarity), while BQL has no such guarantees. Empirically, ABCs demonstrates strong performance when benchmarked across environments drawn from the OpenSpiel game library and OpenAI Gym and exceeds all prior methods in environments which are neither fully stationary nor fully nonstationary.

Easy as ABCs: Unifying Boltzmann Q-Learning and Counterfactual Regret Minimization

TL;DR

ABCs (Adaptive Branching through Child stationarity), a best-of-both-worlds algorithm combining Boltzmann Q-learning (BQL), a classic reinforcement learning algorithm for single-agent domains, and counterfactual regret minimization (CFR), a central algorithm for learning in multi-agent domains, demonstrates strong performance.

Abstract

We propose ABCs (Adaptive Branching through Child stationarity), a best-of-both-worlds algorithm combining Boltzmann Q-learning (BQL), a classic reinforcement learning algorithm for single-agent domains, and counterfactual regret minimization (CFR), a central algorithm for learning in multi-agent domains. ABCs adaptively chooses what fraction of the environment to explore each iteration by measuring the stationarity of the environment's reward and transition dynamics. In Markov decision processes, ABCs converges to the optimal policy with at most an O(A) factor slowdown compared to BQL, where A is the number of actions in the environment. In two-player zero-sum games, ABCs is guaranteed to converge to a Nash equilibrium (assuming access to a perfect oracle for detecting stationarity), while BQL has no such guarantees. Empirically, ABCs demonstrates strong performance when benchmarked across environments drawn from the OpenSpiel game library and OpenAI Gym and exceeds all prior methods in environments which are neither fully stationary nor fully nonstationary.
Paper Structure (56 sections, 14 theorems, 25 equations, 3 figures, 3 tables, 4 algorithms)

This paper contains 56 sections, 14 theorems, 25 equations, 3 figures, 3 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Markov Decision Processes
  5. Finite Extensive-Form Games
  6. Unifying Q-Learning and Counterfactual Regret Minimization
  7. Boltzmann Q-Learning
  8. Counterfactual Regret Minimization
  9. External Sampling MCCFR with Hedge
  10. BQL vs. CFR
  11. Learning our ABCs
  12. Child Stationarity
  13. Detecting Child Stationarity
  14. The ABCs Algorithm
  15. Where ABCs Runs Updates
  16. ...and 41 more sections

Key Result

Theorem 4.2

If $s, a$ satisfy child stationarity with respect to $\sigma, \sigma'$, then $G^\sigma(s, a) = G^{\sigma'}(s, a)$. Furthermore, if $\pi^*$ is a Nash equilibrium of $G^\sigma(s, a)$, then $\pi^*$ is also a Nash equilibrium of $G^{\sigma'}(s, a)$.

Figures (3)

  • Figure 1: ABCs matches the performance of BQL on stationary environments like Cartpole (a) and the performance of CFR methods across several non-stationary environments including weighted rock-paper-scissors (b), Kuhn poker (c), and Leduc poker (d). In a partially stationary environment with elements of both stationarity and nonstationarity, ABCs outperforms both BQL and CFR, being the only algorithm capable of efficiently solving both the Cartpole (e) and Leduc poker (f) portion of the stacked environment.
  • Figure :
  • Figure :

Theorems & Definitions (29)

  • Definition 4.1: Child stationarity
  • Theorem 4.2
  • proof
  • Theorem 4.3
  • proof
  • Theorem 4.4
  • Theorem 4.5
  • Lemma 3.1
  • proof
  • Lemma 3.2
  • ...and 19 more