Evaluating Language Models' Evaluations of Games

TL;DR

This work tackles the meta-question of how AI systems evaluate problems and games, not just solve them. It introduces a formal framework for evaluating AI evaluations and an empirical study using 121 novel two-player games with around 450 human judgments to compare language and reasoning models against people and baselines on payoff and funness. It finds that reasoning models generally align better with human judgments and with game-theoretic payoffs than non-reasoning models, but alignment can be non-monotonic as models become more optimal. It also reveals substantial variability in reasoning resource usage and greater difficulty in quantifying fun, motivating resource-rational meta-reasoning and broader evaluation of AI cognition.

Abstract

Reasoning is not just about solving problems -- it is also about evaluating which problems are worth solving at all. Evaluations of artificial intelligence (AI) systems primarily focused on problem solving, historically by studying how models play games such as chess and Go. In this paper, we advocate for a new paradigm that assesses AI systems' evaluation of games. First, we introduce a formalism for evaluating such evaluations. We then leverage a large-scale dataset of over $100$ novel board games and over 450 human judgments to compare evaluations produced by modern language and reasoning models against those of people and symbolic computational agents. We consider two kinds of evaluative queries: assessing the payoff (or fairness) and the funness of games. These queries span two dimensions relevant to the design of evaluations of AI evaluations: how complex a query is to compute and how difficult a query is to quantify. Our results show that reasoning models are generally more aligned to people in their evaluations of games than non-reasoning language models. However, we observe a non-monotonic relationship: as models get closer to game-theoretic optimal, their fit to human data weakens. We also observe more "jaggedness" across models for assessing funness, in line with the greater difficulty of quantifying this query. Across queries and games, reasoning models show highly variable and unpredictable resource usage when assessing queries, pointing to the importance of imbuing more resource-rational meta-reasoning in language and reasoning models.

Figures (18)

• Figure 1: Evaluating AI systems' evaluations. a, A holistic understanding of model reasoning demands not just assessing how AI systems solve problems (play games), but how they evaluate whether problems, systems, or games are worth pursuing at all; b, Not all evaluations of problems are interesting for evaluating models. Good evaluation queries pose a challenge by being difficult to compute, difficult to quantify, or both.
• Figure 2: Evaluating payoff (fairness) evaluations. a,$R^2$ between human- and model-predicted payoff evaluations, over all $121$ games. Each cell reports the $R^2$ in payoff evaluations between two reasoners. b, Payoff predictions across a subset of the OpenAI model family, compared to people's predicted payoffs (blue) and the estimated game-theoretic optimal (grey). Error bars depict bootstrapped $R^2$ 95% CIs. c-d, Example human- and a subset of model-predicted game evaluations. The distribution over human participants' judgments or each models' $20$ rollouts are shown; the vertical axis shows the normalized density over binned distributions. Non-reasoning models (GPT-4 and DeepSeek-v3) are prompted with CoT. c, depicts a game where reasoning models are more aligned to people's evaluations; in other games as in d, judgments are highly varied across models---with no model faithfully capturing the rich structure in the distribution of human judgments. More example games are included in Appendix \ref{['sec:addtl-analyses']}.
• Figure 3: Evaluating funness evaluations. a,$R^2$ between human- and model-predicted funness evaluations, over all $121$ games. Each cell reports the $R^2$ in funness evaluations between those two reasoners. b, Funness evaluations across a subset of the OpenAI model family reveals non-monotonicty in fits when moving from non-reasoning to reasoning models. Bootstrapped $R^2$ are computed relative to people's predicted funness, with error bars depicting the bootstrapped 95% CIs. c-d Example human- and model-predicted distributions over funness. The vertical axis shows the normalized density over the histogram of people and models' binned judgments. c, depicts an example where people and most models (though not all, e.g., o1) recognize that the game is unlikely to be fun; d, however, people's funness judgments are also variable, e.g., showing bimodality. This bimodality is not captured by most models, with a few exceptions (e.g., GPT-5) for this game. More example game evaluations are in Appendix \ref{['sec:addtl-analyses']}.
• Figure 4: Reasoning tokens used across games and evaluation queries.a,$R^2$ between models' median number of reasoning tokens used per game, for the payoff and funness evaluation queries. b, Median reasoning tokens used for games based on how many "traits" they differ from Tic-Tac-Toe (e.g., a game that is not played on a $3 \times 3$ board, requires $4$ pieces in a row to win, and constrains the win conditions to "only diagonals count," has $3$ traits different from Tic-Tac-Toe). Tic-Tac-Toe itself is zero. The heights of bars show averaged number of median tokens for that game, with error bars depicting standard deviation over games. c, Token usage based on higher-level game category.
• Figure 5: Distribution over models' and people's predicted payoff judgments for example games. Example hand-selected representative game evaluations. The distribution over human participants' judgments or each models' $20$ rollouts are shown. Panels a and d show the complete set of models for the examples shown in Figure \ref{['fig:payoff-preds']}.