MastermindEval: A Simple But Scalable Reasoning Benchmark

TL;DR

MastermindEval presents a simple, scalable benchmark for evaluating deductive reasoning in large language models by simulating Mastermind-like puzzles. It introduces two primary evaluation paradigms—agentic play and deductive reasoning—plus a multiple-choice variant, all supported by an open-source framework. Empirical results across open-source and proprietary models show that while model size improves performance, multi-step deduction remains challenging as task complexity increases, and test-time compute adjusts with task demands. The work provides a principled, extensible way to separate genuine reasoning from mere strategic play, and highlights directions for reasoning-focused model training and evaluation.

Abstract

Recent advancements in large language models (LLMs) have led to remarkable performance across a wide range of language understanding and mathematical tasks. As a result, increasing attention has been given to assessing the true reasoning capabilities of LLMs, driving research into commonsense, numerical, logical, and qualitative reasoning. However, with the rapid progress of reasoning-focused models such as OpenAI's o1 and DeepSeek's R1, there has been a growing demand for reasoning benchmarks that can keep pace with ongoing model developments. In this paper, we introduce MastermindEval, a simple, scalable, and interpretable deductive reasoning benchmark inspired by the board game Mastermind. Our benchmark supports two evaluation paradigms: (1) agentic evaluation, in which the model autonomously plays the game, and (2) deductive reasoning evaluation, in which the model is given a pre-played game state with only one possible valid code to infer. In our experimental results we (1) find that even easy Mastermind instances are difficult for current models and (2) demonstrate that the benchmark is scalable to possibly more advanced models in the future Furthermore, we investigate possible reasons why models cannot deduce the final solution and find that current models are limited in deducing the concealed code as the number of statement to combine information from is increasing.

Figures (4)

• Figure 1: (Left) A Mastermind game instance with a secret code of length 4 and 6 possible colors, along with corresponding feedback for each guess (black and white pegs). (Center) Our strategy evaluation paradigm in MastermindEval, where the LLM acts as the codebreaker in a multi-turn chat environment. (Right) Using Knuth's Five-Guess Algorithm, we pre-play games until only one valid code remains, then prompt the model to deduce the final answer based on the provided hints.
• Figure 2: Test-time compute increases as the model complexity increases indicating MastermindEval can be easily scaled by increasing the length of the secret code $c$ and the number of possible colors/symbols $n$.
• Figure 3: We cluster the solve rate based on the number of hints required to deduce the secret code. We observe that the solve rate approaches 0 when the final code can only be deduced after four chat interactions, even for capable models such as GPT-4o.
• Figure 4: The initial prompt used to play a game of Mastermind. We append each answer of the LLM to the prompt such that LLM are enabled to reflect on their own thinking in future steps.