Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Equilibrium Refinements Improve Subgame Solving in Imperfect-Information Games

Ondrej Kubicek, Viliam Lisy, Tuomas Sandholm

TL;DR

This work addresses subgame solving in imperfect-information games by introducing gadget game sequential equilibria ($GGSE$), a refinement of gadget-game Nash equilibria that preserves rational play at the subgame boundary. It provides practical algorithms to converge to $GGSE$ using minimal overhead: a perturbation of the sequence-form linear program and a gadget-aware CFR framework with trembles, guided by a blueprint prior. The authors demonstrate that different gadget equilibria yield substantially different exploitability in the full game, showing that resolving gadget games can outperform max-margin ones under realistic priors, and that the proposed methods reliably reduce exploitability by substantial margins across benchmark games. This approach offers a safer, more robust subgame solving paradigm with real-world applicability to large imperfect-information domains. The work also outlines avenues for future research, including new gadget variants, full gadget-like constructions, and extensions to large-scale or real-world games such as Texas Hold'em and dark chess.

Abstract

Subgame solving is a technique for scaling algorithms to large games by locally refining a precomputed blueprint strategy during gameplay. While straightforward in perfect-information games where search starts from the current state, subgame solving in imperfect-information games must account for hidden states and uncertainty about the opponent's past strategy. Gadget games were developed to ensure that the improved subgame strategy is robust against any possible opponent's strategy in a zero-sum game. Gadget games typically contain infinitely many Nash equilibria. We demonstrate that while these equilibria are equivalent in the gadget game, they yield vastly different performance in the full game, even when facing a rational opponent. We propose gadget game sequential equilibria as the preferred solution concept. We introduce modifications to the sequence-form linear program and counterfactual regret minimization that converge to these refined solutions with only mild additional computational cost. Additionally, we provide several new insights into the surprising superiority of the resolving gadget game over the max-margin gadget game. Our experiments compare different Nash equilibria of gadget games in several standard benchmark games, showing that our refined equilibria consistently outperform unrefined Nash equilibria, and can reduce the exploitability of the overall strategy by more than 50%

Equilibrium Refinements Improve Subgame Solving in Imperfect-Information Games

TL;DR

This work addresses subgame solving in imperfect-information games by introducing gadget game sequential equilibria (), a refinement of gadget-game Nash equilibria that preserves rational play at the subgame boundary. It provides practical algorithms to converge to using minimal overhead: a perturbation of the sequence-form linear program and a gadget-aware CFR framework with trembles, guided by a blueprint prior. The authors demonstrate that different gadget equilibria yield substantially different exploitability in the full game, showing that resolving gadget games can outperform max-margin ones under realistic priors, and that the proposed methods reliably reduce exploitability by substantial margins across benchmark games. This approach offers a safer, more robust subgame solving paradigm with real-world applicability to large imperfect-information domains. The work also outlines avenues for future research, including new gadget variants, full gadget-like constructions, and extensions to large-scale or real-world games such as Texas Hold'em and dark chess.

Abstract

Subgame solving is a technique for scaling algorithms to large games by locally refining a precomputed blueprint strategy during gameplay. While straightforward in perfect-information games where search starts from the current state, subgame solving in imperfect-information games must account for hidden states and uncertainty about the opponent's past strategy. Gadget games were developed to ensure that the improved subgame strategy is robust against any possible opponent's strategy in a zero-sum game. Gadget games typically contain infinitely many Nash equilibria. We demonstrate that while these equilibria are equivalent in the gadget game, they yield vastly different performance in the full game, even when facing a rational opponent. We propose gadget game sequential equilibria as the preferred solution concept. We introduce modifications to the sequence-form linear program and counterfactual regret minimization that converge to these refined solutions with only mild additional computational cost. Additionally, we provide several new insights into the surprising superiority of the resolving gadget game over the max-margin gadget game. Our experiments compare different Nash equilibria of gadget games in several standard benchmark games, showing that our refined equilibria consistently outperform unrefined Nash equilibria, and can reduce the exploitability of the overall strategy by more than 50%
Paper Structure (25 sections, 5 theorems, 17 equations, 21 figures, 1 table)

This paper contains 25 sections, 5 theorems, 17 equations, 21 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Notation
  3. Subgame Solving
  4. Example
  5. Gadget Game Sequential Equilibria
  6. Choice of Prior
  7. Superiority of the Resolving Gadget Game
  8. Converging to Gadget Game Sequential Equilibria
  9. Sequence-Form Linear Program
  10. Counterfactual Regret Minimization
  11. Experiments
  12. Nested Depth-Limited Subgame Solving
  13. Conclusions and Future Research
  14. Theorems and Observations
  15. Additional Experimental Details
  16. ...and 10 more sections

Key Result

Theorem 1

Given a blueprint strategy $\overline{\pi_{i}}$, a subgame $\mathcal{G}^{s_{0}}{}$, and a subgame strategy $\pi_{i}^{\mathcal{G}^{s_{0}}{}}$, let $\pi_{i} = \overline{\pi_{i}} \gets \pi_{i}^{\mathcal{G}^{s_{0}}{}}$. Let $\overline{\pi_{-i}^{BR}} \in BR_{-i}(\overline{\pi_{i}})$ and $\pi_{-i}^{BR} \i

Figures (21)

  • Figure 1: Example game, that highlights the problem with sequential rationality in subgame solving. Player 1 (red) is maximizing the value, while Player 2 (blue) is minimizing. Chance player (grey) is playing according to a shown fixed "probability" distribution.
  • Figure 2: Exploitability of the resolved strategy from each subgame based on blueprint's exploitability when using CFR as solver.
  • Figure 3: CFR Convergence curves of subgame solving techniques.
  • Figure 4: Head-to-head win-rate of each subgame solving technique against its 'worst-matchup' opponent out of the other techniques.
  • Figure 5: An example of a game, where different equilibria in gadget games result in different utility
  • ...and 16 more figures

Theorems & Definitions (13)

  • Definition 1
  • Theorem 1
  • Proposition 1
  • Definition 2
  • Theorem 2
  • Definition 3
  • Theorem 2
  • proof
  • proof
  • Theorem 2
  • ...and 3 more