GameArena: Evaluating LLM Reasoning through Live Computer Games

TL;DR

GameArena presents a dynamic, human-in-the-loop benchmark to evaluate fine-grained LLM reasoning through three interactive games (Akinator, Taboo, Bluffing). By constraining interactions with game rules and collecting retrospective, step-by-step reasoning data, it enables measurement of deductive, inductive, abductive, and multi-hop reasoning across multiple state-of-the-art models. The study demonstrates superior data efficiency and user engagement versus prior dynamic benchmarks, and shows strong correlations with established reasoning benchmarks while highlighting domain-specific differences. The work provides a scalable framework for expanding the game suite and refining reasoning metrics, with an emphasis on releasing the gaming data for future research and a robust ethics protocol for human participants.

Abstract

Evaluating the reasoning abilities of large language models (LLMs) is challenging. Existing benchmarks often depend on static datasets, which are vulnerable to data contamination and may get saturated over time, or on binary live human feedback that conflates reasoning with other abilities. As the most prominent dynamic benchmark, Chatbot Arena evaluates open-ended questions in real-world settings, but lacks the granularity in assessing specific reasoning capabilities. We introduce GameArena, a dynamic benchmark designed to evaluate LLM reasoning capabilities through interactive gameplay with humans. GameArena consists of three games designed to test specific reasoning capabilities (e.g., deductive and inductive reasoning), while keeping participants entertained and engaged. We analyze the gaming data retrospectively to uncover the underlying reasoning processes of LLMs and measure their fine-grained reasoning capabilities. We collect over 2000 game sessions and provide detailed assessments of various reasoning capabilities for five state-of-the-art LLMs. Our user study with 100 participants suggests that GameArena improves user engagement compared to Chatbot Arena. For the first time, GameArena enables the collection of step-by-step LLM reasoning data in the wild.

Figures (11)

• Figure 1: In the Akinator game, a player thinks of an object, and the LLM asks a series of yes-or-no questions to guess the object.
• Figure 2: In the Taboo game, a player has a taboo word that they need to prompt the LLM to say without revealing what the word is.
• Figure 3: In the Bluffing game, the player makes a statement and responds to a series of questions from the LLM as if the statement is true, trying to trick the LLM into believing it.
• Figure 4: Participant rating distributions across different games in GameArena and Chatbot Arena. A lower bar represents a lower proportion of participants expressing a positive attitude. GameArena showed higher user enjoyment, satisfaction and willingness to participate than Chatbot Arena.
• Figure 5: Controlled experiment results for Akinator and Bluffing. The numerical values are averaged across the five different system prompts, with the standard deviation included as the error bar.