Evaluating Large Language Models with Grid-Based Game Competitions: An Extensible LLM Benchmark and Leaderboard

TL;DR

The paper addresses the problem of evaluating LLMs' strategic reasoning and rule adherence beyond traditional NLP benchmarks. It introduces an extensible grid-game benchmark with Tic-Tac-Toe, Connect Four, and Gomoku, paired with three prompt types and a web-based simulator that records rich per-move data, producing outputs in JSON, CSV, TXT, and PNG for a GitHub-hosted leaderboard. Key contributions include 2,310 matches across seven LLMs plus a random opponent, a modular data pipeline, and open data to encourage community participation and new games. The work highlights significant variability in LLM performance across games and prompt formats, underlines challenges in handling visual prompts, and provides a practical framework to advance benchmarking toward more robust strategic reasoning in AI systems.

Abstract

We introduce a novel and extensible benchmark for large language models (LLMs) through grid-based games such as Tic-Tac-Toe, Connect Four, and Gomoku. The open-source game simulation code, available on GitHub, allows LLMs to compete and generates detailed data files in JSON, CSV, TXT, and PNG formats for leaderboard rankings and further analysis. We present the results of games among leading LLMs, including Claude 3.5 Sonnet and Claude 3 Sonnet by Anthropic, Gemini 1.5 Pro and Gemini 1.5 Flash by Google, GPT-4 Turbo and GPT-4o by OpenAI, and Llama3-70B by Meta. We also encourage submissions of results from other LLMs. In total, we simulated 2,310 matches (5 sessions for each pair among 7 LLMs and a random player) across three types of games, using three distinct prompt types: list, illustration, and image. The results revealed significant variations in LLM performance across different games and prompt types, with analysis covering win and disqualification rates, missed opportunity analysis, and invalid move analysis. The details of the leaderboard and result matrix data are available as open-access data on GitHub. This study enhances our understanding of LLMs' capabilities in playing games they were not specifically trained for, helping to assess their rule comprehension and strategic thinking. On the path to Artificial General Intelligence (AGI), this study lays the groundwork for future exploration into their utility in complex decision-making scenarios, illuminating their strategic thinking abilities and offering directions for further inquiry into the limits of LLMs within game-based frameworks.

Figures (24)

• Figure 1: Web-based app for game simulation shows the progress of a Connect Four game.
• Figure 2: The illustration of web-based app and web service interactions to play a game.
• Figure 3: Tic-Tac-Toe game outcomes using the ‘list’ prompt where each LLM faced six others and the ‘random play’ as both player 1 and player 2, playing each opponent 5 times (280 games total).
• Figure 4: Tic-Tac-Toe game outcomes using the ‘illustration’ prompt where each LLM faced six others and the ‘random play’ as both player 1 and player 2, playing each opponent 5 times (280 games total).
• Figure 5: Tic-Tac-Toe game outcomes using the ‘image’ prompt where each LLM faced five others and the ‘random play’ as both player 1 and player 2, playing each opponent 5 times (210 games total).