Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Yuzhe Tang

Abstract

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.

BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models

Abstract

Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds. We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories, each targeting a specific commonsense reasoning failure mode in LLMs. Categories range from implicit physical constraints ("Should I walk or drive my rental car to the return lot?") to semantic scope tricks and default assumption hijacks. We evaluate eight frontier models -- four from the Claude family and four from the GPT family -- using a zero-shot protocol with 10 independent runs per question. The best model, Claude Opus 4.6 with extended thinking, achieves only 80.3% accuracy; the worst, GPT-4o, scores 39.7%. Even top-performing models exhibit a 6-16 percentage-point gap between accuracy and consistency, revealing stochastic reasoning. Cross-lingual evaluation in Chinese shows most models degrade by 2-8 percentage points, confirming that these failures reflect reasoning deficits rather than language-specific artifacts. BrainBench provides a fine-grained diagnostic tool for identifying where and why LLMs substitute surface heuristics for genuine commonsense reasoning.
Paper Structure (41 sections, 4 figures, 6 tables)

This paper contains 41 sections, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Commonsense reasoning benchmarks.
  4. The Car Wash Test.
  5. BRAINTEASER and lateral thinking.
  6. SimpleBench.
  7. LLM reasoning and chain-of-thought.
  8. The BrainBench Dataset
  9. Category Design
  10. Question Design
  11. Illustrative Examples
  12. Example 1: Implicit physical constraint.
  13. Example 2: Broken/dead device self-reference.
  14. Example 3: Default assumption hijack.
  15. Experimental Setup
  16. ...and 26 more sections

Figures (4)

  • Figure 1: Overall accuracy (%) on BrainBench (English). Models are ranked by accuracy. Error bars indicate the range between accuracy and consistency, capturing reasoning reliability.
  • Figure 2: Accuracy heatmap across models and categories. Categories are sorted by average difficulty (hardest at left). Darker cells indicate lower accuracy.
  • Figure 3: Accuracy comparison between standard and extended-thinking variants by category. Green bars indicate categories where thinking helps; red bars indicate where it hurts.
  • Figure 4: Accuracy comparison between English and Chinese versions of BrainBench.