Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Do You Get the Hint? Benchmarking LLMs on the Board Game Concept

Ine Gevers, Walter Daelemans

TL;DR

This paper presents Concept, a natural-language board game designed to probe abductive and abstract reasoning in LLMs. By collecting human and model performance on English, Dutch, French, and Spanish clue-guess rounds, the authors show humans reliably solve concepts (>90% success) while state-of-the-art LLMs struggle (below 40%), particularly in inferring players' strategic intents and updating hypotheses as clues evolve. They compare static and dynamic prompting, finding static prompts generally superior and dynamic interaction insufficient for improving reasoning, even for a reasoning-leaning model. The multilingual results reveal weaker transfer to non-English languages, highlighting limitations in cross-language abstraction and transfer. The work provides a public dataset, evaluates a broad set of models, and points to future directions such as fine-tuning, interpretability studies, and multi-agent extensions to enhance abductive reasoning in language models.

Abstract

Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In particular, tasks that require abstract reasoning remain challenging, often because they use representations such as grids, symbols, or visual patterns that differ from the natural language data LLMs are trained on. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning in a representation that is much closer to LLM pre-training data: natural language. Our results show that this game, easily solved by humans (with a success rate of over 90\%), is still very challenging for state-of-the-art LLMs (no model exceeds 40\% success rate). Specifically, we observe that LLMs struggle with interpreting other players' strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.

Do You Get the Hint? Benchmarking LLMs on the Board Game Concept

TL;DR

This paper presents Concept, a natural-language board game designed to probe abductive and abstract reasoning in LLMs. By collecting human and model performance on English, Dutch, French, and Spanish clue-guess rounds, the authors show humans reliably solve concepts (>90% success) while state-of-the-art LLMs struggle (below 40%), particularly in inferring players' strategic intents and updating hypotheses as clues evolve. They compare static and dynamic prompting, finding static prompts generally superior and dynamic interaction insufficient for improving reasoning, even for a reasoning-leaning model. The multilingual results reveal weaker transfer to non-English languages, highlighting limitations in cross-language abstraction and transfer. The work provides a public dataset, evaluates a broad set of models, and points to future directions such as fine-tuning, interpretability studies, and multi-agent extensions to enhance abductive reasoning in language models.

Abstract

Large language models (LLMs) have achieved striking successes on many benchmarks, yet recent studies continue to expose fundamental weaknesses. In particular, tasks that require abstract reasoning remain challenging, often because they use representations such as grids, symbols, or visual patterns that differ from the natural language data LLMs are trained on. In this paper, we introduce Concept, a simple word-guessing board game, as a benchmark for probing abductive reasoning in a representation that is much closer to LLM pre-training data: natural language. Our results show that this game, easily solved by humans (with a success rate of over 90\%), is still very challenging for state-of-the-art LLMs (no model exceeds 40\% success rate). Specifically, we observe that LLMs struggle with interpreting other players' strategic intents, and with correcting initial hypotheses given sequential information updates. In addition, we extend the evaluation across multiple languages, and find that the LLM performance drops further in lower-resource languages (Dutch, French, and Spanish) compared to English.
Paper Structure (10 sections, 3 figures, 5 tables)

This paper contains 10 sections, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Data
  5. Prompting
  6. Evaluation
  7. Results
  8. Conclusion
  9. Model specifications
  10. Prompt

Figures (3)

  • Figure 1: A simplified example of a round of Concept, comparing guesses made by humans and an LLM.
  • Figure 2: Distribution of number of guesses by humans.
  • Figure 3: Comparison of how many guesses were needed to arrive at the correct concept, for concepts that are found both by humans and the LLM.