Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RiddleBench: A New Generative Reasoning Benchmark for LLMs

Deepon Halder, Alan Saji, Thanmay Jayakumar, Ratish Puduppully, Anoop Kunchukuttan, Raj Dabre

TL;DR

RiddleBench introduces a diagnostic benchmark of 1,737 English puzzles designed to probe multifaceted reasoning that combines logical deduction, spatial awareness, and constraint satisfaction. The authors evaluate a broad set of LLMs under zero-shot prompting and analyze not only final accuracy but also reasoning reliability, including cross-model correction, self-correction, and robustness to prompt perturbations. Key findings reveal a hallucination cascade when models evaluate others’ reasoning, a strong self-confirmation bias that hampers self-correction, and fragile reasoning that degrades with constraint reordering or irrelevant information. Overall, RiddleBench exposes critical weaknesses in current LLM reasoning and provides a foundation for developing more robust, interpretable, and trustworthy systems, with plans for expansion to more languages and broader data collection.

Abstract

Large Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models.

RiddleBench: A New Generative Reasoning Benchmark for LLMs

TL;DR

RiddleBench introduces a diagnostic benchmark of 1,737 English puzzles designed to probe multifaceted reasoning that combines logical deduction, spatial awareness, and constraint satisfaction. The authors evaluate a broad set of LLMs under zero-shot prompting and analyze not only final accuracy but also reasoning reliability, including cross-model correction, self-correction, and robustness to prompt perturbations. Key findings reveal a hallucination cascade when models evaluate others’ reasoning, a strong self-confirmation bias that hampers self-correction, and fragile reasoning that degrades with constraint reordering or irrelevant information. Overall, RiddleBench exposes critical weaknesses in current LLM reasoning and provides a foundation for developing more robust, interpretable, and trustworthy systems, with plans for expansion to more languages and broader data collection.

Abstract

Large Language Models have demonstrated strong performance on many established reasoning benchmarks. However, these benchmarks primarily evaluate structured skills like quantitative problem-solving, leaving a gap in assessing flexible, multifaceted reasoning abilities that are central to human intelligence. These abilities require integrating logical deduction with spatial awareness and constraint satisfaction, which current evaluations do not measure well. To address this, we introduce RiddleBench, a benchmark of 1,737 challenging puzzles in English designed to probe these core reasoning capabilities. Evaluation of state-of-the-art models on RiddleBench shows fundamental weaknesses. Even top proprietary models like Gemini 2.5 Pro, o3, and Claude 4 Sonnet achieve accuracy just above 60% (60.30%, 63.37%, and 63.16%). Analysis further reveals deep failures, including hallucination cascades (accepting flawed reasoning from other models) and poor self-correction due to a strong self-confirmation bias. Their reasoning is also fragile, with performance degrading significantly when constraints are reordered or irrelevant information is introduced. RiddleBench functions as a diagnostic tool for these issues and as a resource for guiding the development of more robust and reliable language models.

Paper Structure

This paper contains 32 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. RiddleBench
  4. Data Collection and Curation
  5. Experimental Design
  6. Results and Analyses
  7. Overall Performance
  8. The 'Hallucination Cascade': Failures in Cross-Model Correction
  9. The Illusion of Self-Correction
  10. Fragile Reasoning: The Lack of Robustness
  11. Conclusion
  12. Limitations
  13. Future Work
  14. Ethics Statement
  15. Acknowledgements
  16. ...and 17 more sections

Figures (6)

  • Figure 1: An example from the RiddleBench benchmark for Blood Relations.
  • Figure 2: The step-by-step methodology for building RiddleBench. The workflow combines automated extraction with meticulous human evaluation to ensure high-quality data.
  • Figure 3: A radar chart illustrating the performance of evaluated LLMs across the four reasoning categories of RiddleBench. Each colored line represents a different model, showing its strengths and weaknesses in SR, SA, BR, and CD.
  • Figure 4: Overall performance of evaluated LLMs on RiddleBench. The horizontal bars indicate the percentage of correct answers for each model.
  • Figure 5: An ASCII family tree generated by a Gemini model for a Blood Relations puzzle, a unique visual reasoning strategy observed in our analysis.
  • ...and 1 more figures