Combining Counterfactual Regret Minimization with Information Gain to Solve Extensive Games with Unknown Environments

TL;DR

A method that combines CFR with information gain to compute the Nash equilibrium of IIEGs with unknown environments and significantly reduces the number of interactions with the environment compared to the different baselines and computes a more accurate approximate NE within the same number of interaction rounds.

Abstract

Counterfactual regret minimization (CFR) is an effective algorithm for solving extensive games with imperfect information (IIEGs). However, CFR is only allowed to be applied in known environments, where the transition function of the chance player and the reward function of the terminal node in IIEGs are known. In uncertain situations, such as reinforcement learning (RL) problems, CFR is not applicable. Thus, applying CFR in unknown environments is a significant challenge that can also address some difficulties in the real world. Currently, advanced solutions require more interactions with the environment and are limited by large single-sampling variances to narrow the gap with the real environment. In this paper, we propose a method that combines CFR with information gain to compute the Nash equilibrium (NE) of IIEGs with unknown environments. We use a curiosity-driven approach to explore unknown environments and minimize the discrepancy between uncertain and real environments. Additionally, by incorporating information into the reward, the average strategy calculated by CFR can be directly implemented as the interaction policy with the environment, thereby improving the exploration efficiency of our method in uncertain environments. Through experiments on standard testbeds such as Kuhn poker and Leduc poker, our method significantly reduces the number of interactions with the environment compared to the different baselines and computes a more accurate approximate NE within the same number of interaction rounds.

Figures (3)

• Figure 1: An architecture illustration of our proposed method. The model starts by initializing the data pool with the distribution of rewards. The data pool stores the posterior distribution of the reward corresponding to each action. Then the KL divergence for each node can be calculated using the weight distribution of BNN. The reward $r^{\prime}$ that increases the information gain is assigned to the terminal node in the game tree. Finally, the data pool is updated by interacting with the environment using average strategies.
• Figure 2: A flowchart of our method. a: Constructing a complete game tree and performing dirichlet sampling on rewards in the real game environment. The accuracy of sampling improves progressively with continuous updates to the environment. $r_{n}$ represents the sampled reward. b: The data in the data pool primarily consists of posterior distributions of rewards corresponding to each action. $r_{n}^{\prime}$ represents the posterior distribution of reward. c: It shows the BNN used for our work. The one-hot encoding of nodes is input into the BNN, producing posterior samples of rewards. $W_{1}$ and $W_{2}$ represent the weight distributions in the BNN. The KL divergence value for each node in the game tree is obtained through additional computation on the weights of the BNN. d: The value of KL divergence is considered equivalent to information gain, and it is added as the agent's intrinsic reward to the original reward (i.e., $r_{n}+\eta D_{KL}$). This facilitates the agent in generating strategies to continue exploring unknown environments and find an NE.
• Figure 6: The ablation experiments of our methods, VDCFR and VFSP, were conducted in Kuhn poker with unknown environments.