No-Regret Strategy Solving in Imperfect-Information Games via Pre-Trained Embedding

TL;DR

The paper tackles the challenge of solving large-scale imperfect-information extensive-form games under limited resources, where traditional pre-trained clustering-based abstractions lose fine-grained distinctions between information sets.It introduces Embedding CFR, a framework that pre-trains information-set embeddings in a low-dimensional space using an advisor-based extended abstraction and an embedding matrix, enabling regret-based strategy solving in the embedding space with memory-efficient updates.The authors provide an approximate convergence analysis showing regret can decrease within the embedding space when advisors act in isolation, and they design a poker-specific embedding pipeline (HandEbdNet) to generate embedding coordinates on the fly.Empirical results in Numeral211 Hold'em demonstrate that Embedding CFR achieves substantially faster exploitability convergence than state-of-the-art clustering-based abstractions under the same resource constraints, validating the effectiveness of embedding-based information-set abstraction for poker AI.

Abstract

High-quality information set abstraction remains a core challenge in solving large-scale imperfect-information extensive-form games (IIEFGs)--such as no-limit Texas Hold'em--where the finite nature of spatial resources hinders solving strategies for the full game. State-of-the-art AI methods rely on pre-trained discrete clustering for abstraction, yet their hard classification irreversibly discards critical information: specifically, the quantifiable subtle differences between information sets--vital for strategy solving--thus compromising the quality of such solving. Inspired by the word embedding paradigm in natural language processing, this paper proposes the Embedding CFR algorithm, a novel approach for solving strategies in IIEFGs within an embedding space. The algorithm pre-trains and embeds the features of individual information sets into an interconnected low-dimensional continuous space, where the resulting vectors more precisely capture both the distinctions and connections between information sets. Embedding CFR introduces a strategy-solving process driven by regret accumulation and strategy updates in this embedding space, with supporting theoretical analysis verifying its ability to reduce cumulative regret. Experiments on poker show that with the same spatial overhead, Embedding CFR achieves significantly faster exploitability convergence compared to cluster-based abstraction algorithms, confirming its effectiveness. Furthermore, to our knowledge, it is the first algorithm in poker AI that pre-trains information set abstractions via low-dimensional embedding for strategy solving.

Figures (8)

• Figure 1: Behavior comparison of hand representations under Embedding CFR and traditional information set abstraction for hands $\blacksquare$, $\hbox{$\bullet$}$, and $\hbox{$\blacksquare$}$ in Texas Hold'em. (a) Embedding CFR maps hands to embedding coordinates, which form an m-dimensional probability distribution where the sum of values across all dimensions equals 1. (b) Schematic 2D projection of embedding coordinates illustrates the geometric topology between hands, highlighting both similarity (closeness of $\blacksquare$ and $\hbox{$\bullet$}$) and distinction (separation from $\hbox{$\blacksquare$}$). (c) Traditional abstraction maps information sets to a fixed number of $m$ abstracted classes, e.g. buckets, forcing binary decisions for these hands: either refining $\blacksquare$ and $\hbox{$\bullet$}$ into distinct equivalence classes or coarsening them into one. This lack of intermediate states hinders exploitation of inter-information-set similarity for strategy solving.
• Figure 2: Schematic comparison of driving processes: Embedding CFR vs. Vanilla CFR
• Figure 3: Two info-block partitioning schemes for player 1's infosets $I_1, I_2, I_3$ in Kuhn Poker, noting that all sets satisfy action-space consistency ($A(I_1) = A(I_2) = A(I_3)$).
• Figure 4: The network architecture of HandEbdNet.
• Figure 5: Exploitability convergence comparison of clustering-based algorithm (EHS, PaEmd, KrwEmd) vs. Embedding CFR algorithms in Numeral211 Hold'em.

Theorems & Definitions (7)

• Definition 3.1: Extended Information Set Abstraction
• Proposition 1
• Lemma 1: lanctot2009monte, Lemma 7
• proof
• Lemma 2
• proof
• proof : Proof of Proposition \ref{['thm:regret_decrease']}