PerfectDou: Dominating DouDizhu with Perfect Information Distillation

TL;DR

The paper tackles imperfect-information multi-agent decision problems by introducing PerfectDou, a DouDizhu AI built with a perfect information distillation (PTIE) training paradigm. PTIE trains value estimates with access to perfect information $D(h)$ while deploying policies that operate under imperfect information $h$, integrated into a centralized-training/decentralized-execution scheme and optimized with proximal policy optimization (PPO) and generalized advantage estimation (GAE). Empirically, PerfectDou achieves state-of-the-art performance, dominating prior DouDizhu AIs such as DouZero and demonstrating superior sample efficiency and robust cooperation between the two Peasants. The authors also design rich card/node representations and a oracle-based node reward to guide training, showing that distilling perfect information into the training process yields more rational and strategic play, with potential applicability to other imperfect-information domains.

Abstract

As a challenging multi-player card game, DouDizhu has recently drawn much attention for analyzing competition and collaboration in imperfect-information games. In this paper, we propose PerfectDou, a state-of-the-art DouDizhu AI system that dominates the game, in an actor-critic framework with a proposed technique named perfect information distillation. In detail, we adopt a perfect-training-imperfect-execution framework that allows the agents to utilize the global information to guide the training of the policies as if it is a perfect information game and the trained policies can be used to play the imperfect information game during the actual gameplay. To this end, we characterize card and game features for DouDizhu to represent the perfect and imperfect information. To train our system, we adopt proximal policy optimization with generalized advantage estimation in a parallel training paradigm. In experiments we show how and why PerfectDou beats all existing AI programs, and achieves state-of-the-art performance.

Figures (8)

• Figure 1: Overview of perfect information distillation within a perfect-training-imperfect-execution framework. The value network takes additional information (such as other players' cards in Poker games) as input, while the policy network does not.
• Figure 2: Card representation matrix. Columns stand for 15 different card ranks and rows stand for correspondingly designed features. The first 4 rows are the same as Zha et al. zhadouzero21, and the last 8 rows are additional design for encoding the legal combination of cards.
• Figure 3: The policy network structure of PerfectDou system. The network predicts the action distribution given the current imperfect information of the game, including state information and available actions feature.
• Figure 4: Illustration of the distributed training system.
• Figure 5: Comparison of the inference time.