Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenGuanDan: A Large-Scale Imperfect Information Game Benchmark

Chao Li, Shangdong Yang, Chiheng Zhan, Zhenxing Ge, Yujing Hu, Bingkun Bao, Xingguo Chen, Yang Gao

TL;DR

OpenGuanDan introduces a large-scale, imperfect-information GuanDan benchmark with a high-throughput simulator and independent per-player APIs, enabling robust evaluation of learning-based and rule-based agents as well as human-AI interactions. The framework emphasizes challenges such as partial observability, expansive action spaces, mixed cooperative-competitive objectives, long-horizon reasoning, and dynamic team composition, and demonstrates a throughput of approximately $25$ million steps per hour on a 10-environment setup. Empirical results show learning-based agents (notably GS2, SDMC, and DanZero) outperform rule-based baselines, yet none reach superhuman performance, underscoring significant room for methodological advances in multi-agent decision-making. The work provides a practical, extensible platform for rigorous comparison and future research, with public code and a focus on integrating RL, game theory, and potentially LLM-based approaches to address GuanDan’s core challenges.

Abstract

The advancement of data-driven artificial intelligence (AI), particularly machine learning, heavily depends on large-scale benchmarks. Despite remarkable progress across domains ranging from pattern recognition to intelligent decision-making in recent decades, exemplified by breakthroughs in board games, card games, and electronic sports games, there remains a pressing need for more challenging benchmarks to drive further research. To this end, this paper proposes OpenGuanDan, a novel benchmark that enables both efficient simulation of GuanDan (a popular four-player, multi-round Chinese card game) and comprehensive evaluation of both learning-based and rule-based GuanDan AI agents. OpenGuanDan poses a suite of nontrivial challenges, including imperfect information, large-scale information set and action spaces, a mixed learning objective involving cooperation and competition, long-horizon decision-making, variable action spaces, and dynamic team composition. These characteristics make it a demanding testbed for existing intelligent decision-making methods. Moreover, the independent API for each player allows human-AI interactions and supports integration with large language models. Empirically, we conduct two types of evaluations: (1) pairwise competitions among all GuanDan AI agents, and (2) human-AI matchups. Experimental results demonstrate that while current learning-based agents substantially outperform rule-based counterparts, they still fall short of achieving superhuman performance, underscoring the need for continued research in multi-agent intelligent decision-making domain. The project is publicly available at https://github.com/GameAI-NJUPT/OpenGuanDan.

OpenGuanDan: A Large-Scale Imperfect Information Game Benchmark

TL;DR

OpenGuanDan introduces a large-scale, imperfect-information GuanDan benchmark with a high-throughput simulator and independent per-player APIs, enabling robust evaluation of learning-based and rule-based agents as well as human-AI interactions. The framework emphasizes challenges such as partial observability, expansive action spaces, mixed cooperative-competitive objectives, long-horizon reasoning, and dynamic team composition, and demonstrates a throughput of approximately million steps per hour on a 10-environment setup. Empirical results show learning-based agents (notably GS2, SDMC, and DanZero) outperform rule-based baselines, yet none reach superhuman performance, underscoring significant room for methodological advances in multi-agent decision-making. The work provides a practical, extensible platform for rigorous comparison and future research, with public code and a focus on integrating RL, game theory, and potentially LLM-based approaches to address GuanDan’s core challenges.

Abstract

The advancement of data-driven artificial intelligence (AI), particularly machine learning, heavily depends on large-scale benchmarks. Despite remarkable progress across domains ranging from pattern recognition to intelligent decision-making in recent decades, exemplified by breakthroughs in board games, card games, and electronic sports games, there remains a pressing need for more challenging benchmarks to drive further research. To this end, this paper proposes OpenGuanDan, a novel benchmark that enables both efficient simulation of GuanDan (a popular four-player, multi-round Chinese card game) and comprehensive evaluation of both learning-based and rule-based GuanDan AI agents. OpenGuanDan poses a suite of nontrivial challenges, including imperfect information, large-scale information set and action spaces, a mixed learning objective involving cooperation and competition, long-horizon decision-making, variable action spaces, and dynamic team composition. These characteristics make it a demanding testbed for existing intelligent decision-making methods. Moreover, the independent API for each player allows human-AI interactions and supports integration with large language models. Empirically, we conduct two types of evaluations: (1) pairwise competitions among all GuanDan AI agents, and (2) human-AI matchups. Experimental results demonstrate that while current learning-based agents substantially outperform rule-based counterparts, they still fall short of achieving superhuman performance, underscoring the need for continued research in multi-agent intelligent decision-making domain. The project is publicly available at https://github.com/GameAI-NJUPT/OpenGuanDan.
Paper Structure (19 sections, 4 figures, 3 tables)

This paper contains 19 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. GuanDan: A Chinese Card Game
  4. Game Mechanism
  5. Challenges
  6. Task Formalization
  7. Game Theory Perspective
  8. Reinforcement Learning Perspective
  9. OpenGuanDan
  10. GuanDan Simulator
  11. Built-In Agents
  12. Experiments
  13. Pair-Wise Evaluation
  14. Human-AI Evaluation
  15. Conclusion
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Game complexity of multiple representative card games. We report the information sets (infosets) count, the legal action space size, and the infosets size for Heads-Up Limit Texas Hold'em (HULH), Heads-Up No-Limit Texas Hold'em (HUNLH), Six-Player Texas Hold'em, DouDizhu, Mahjong, and GuanDan. Notably, the information sets size is approximately $10^{3}$ for HULH and HUNLH, $10^{15}$ for Six-Player Texas Hold’em, $10^{23}$ for DouDizhu, $10^{48}$ for Mahjong, and $10^{36}$ for GuanDan.
  • Figure 2: Number of simulation steps per hour versus number of parallel environments for the GuanDan simulator on a machine with 24-core Intel Core i9-13900K CPU (3.00GHz).
  • Figure 3: Visualization of all card combinations and their rankings.
  • Figure 4: A snapshot of the self-developed GuanDan simulator, which supports both Chinese and English.