Efficient Last-iterate Convergence Algorithms in Solving Games
Linjian Meng, Youzhi Zhang, Zhenxing Ge, Shangdong Yang, Tianyu Ding, Wenbin Li, Tianpei Yang, Bo An, Yang Gao
TL;DR
This work addresses the challenge of achieving last-iterate convergence when learning Nash equilibria in extensive-form games. It leverages the Reward Transformation (RT) framework to recast NE learning into a sequence of perturbed regularized EFGs and introduces RTCFR^+, a parameter-free RM-based CFR algorithm that solves these perturbed problems using CFR^+. The authors prove both non-parameter-free and parameter-free last-iterate convergence for CFR^+ in solving perturbed regularized EFGs, enabling end-to-end last-iterate convergence for the original game, and demonstrate superior empirical performance on standard benchmarks. The results offer a practical, tuning-free approach to fast NE computation in large sequential games with strong stability guarantees, supported by theoretical convergence and extensive experiments.
Abstract
To establish last-iterate convergence for Counterfactual Regret Minimization (CFR) algorithms in learning a Nash equilibrium (NE) of extensive-form games (EFGs), recent studies reformulate learning an NE of the original EFG as learning the NEs of a sequence of (perturbed) regularized EFGs. Consequently, proving last-iterate convergence in solving the original EFG reduces to proving last-iterate convergence in solving (perturbed) regularized EFGs. However, the empirical convergence rates of the algorithms in these studies are suboptimal, since they do not utilize Regret Matching (RM)-based CFR algorithms to solve perturbed EFGs, which are known the exceptionally fast empirical convergence rates. Additionally, since solving multiple perturbed regularized EFGs is required, fine-tuning across all such games is infeasible, making parameter-free algorithms highly desirable. In this paper, we prove that CFR$^+$, a classical parameter-free RM-based CFR algorithm, achieves last-iterate convergence in learning an NE of perturbed regularized EFGs. Leveraging CFR$^+$ to solve perturbed regularized EFGs, we get Reward Transformation CFR$^+$ (RTCFR$^+$). Importantly, we extend prior work on the parameter-free property of CFR$^+$, enhancing its stability, which is crucial for the empirical convergence of RTCFR$^+$. Experiments show that RTCFR$^+$ significantly outperforms existing algorithms with theoretical last-iterate convergence guarantees.