Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models

TL;DR

This paper introduces Braingle Brainteaser, a brainteaser benchmark that uses long narrative problems to probe how large language models reason, beyond final-answer accuracy. By decomposing reasoning into semantic parsing, solution generation, self-correction, step breakdown, and hint usage, the study investigates whether LLMs rely on creative insights or brute-force strategies, and how these tendencies shift with model size and prompting. The authors report that larger models and structured prompts can foster more insightful reasoning and reduce brute-force reliance, but that performance still struggles on the hardest problems and with robust self-correction. They also show that translating narratives into formal mathematical statements yields modest gains, and that models can, under some conditions, break down solutions into meaningful steps, though this is not universal. Overall, Braingle Brainteaser challenges common assumptions about reasoning decomposability and highlights the need for systematic approaches to improve creative problem-solving in LLMs, with implications for evaluation, prompting, and future research directions.

Abstract

Accuracy remains a standard metric for evaluating AI systems, but it offers limited insight into how models arrive at their solutions. In this work, we introduce a benchmark based on brainteasers written in long narrative form to probe more deeply into the types of reasoning strategies that models use. Brainteasers are well-suited for this goal because they can be solved with multiple approaches, such as a few-step solution that uses a creative insight or a longer solution that uses more brute force. We investigate large language models (LLMs) across multiple layers of reasoning, focusing not only on correctness but also on the quality and creativity of their solutions. We investigate many aspects of the reasoning process: (1) semantic parsing of the brainteasers into precise mathematical competition style formats; (2) generating solutions from these mathematical forms; (3) self-correcting solutions based on gold solutions; (4) producing step-by-step sketches of solutions; and (5) making use of hints. We find that LLMs are in many cases able to find creative, insightful solutions to brainteasers, suggesting that they capture some of the capacities needed to solve novel problems in creative ways. Nonetheless, there also remain situations where they rely on brute force despite the availability of more efficient, creative solutions, highlighting a potential direction for improvement in the reasoning abilities of LLMs.

Figures (7)

• Figure 1: Distributions of brute-force and non-brute-force solutions. Results are shown for the (a) Math and the (b) Logic datasets. Numbers reflect cases where only the model used brute force.
• Figure 2: Effect of rewriting on the top 30 most difficult rewritable math brainteasers in their original narrative format ("before rewriting") and then rewritten into a more mathematical format ("after rewriting"). (a) Rewriting slightly increases model correctness. (b) For all models, most results remain the same after rewriting, while a small fraction of initially incorrect answers are corrected.
• Figure S1: Correctness on Math categories, using the Math prompt.
• Figure S2: Correctness on Logic categories, using the Math prompt.
• Figure S3: Correctness on Math subcategories, using the Math prompt.