Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems

Stephen Miner, Yoshiki Takashima, Simeng Han, Sam Kouteili, Ferhat Erata, Ruzica Piskac, Scott J Shapiro

TL;DR

The paper addresses the fragility of existing math benchmarks as LLMs approach saturation by proposing Scheherazade, an automated benchmark-generation framework that chains problems via forward and backward reasoning structures. By linking problems with conditional branches and randomness, Scheherazade creates large, nested reasoning tasks from a small seed set and renders scalable evaluation feasible. Evaluations across OpenAI o1, GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 show pronounced declines in both raw and normalized accuracy as chain length increases, with o1-preview exhibiting relatively robust backward reasoning. Error analysis distinguishes semantic understanding failures, wrong-path decisions, false negatives, and rare infinite-loop behavior, providing actionable insights for designing more discriminative benchmarks and understanding model reasoning patterns.

Abstract

Benchmarks are critical for measuring Large Language Model (LLM) reasoning capabilities. Some benchmarks have even become the de facto indicator of such capabilities. However, as LLM reasoning capabilities improve, existing widely-used benchmarks such as GSM8K marginally encapsulate model reasoning differentials - most state-of-the-art models for example achieve over 94% accuracy on the GSM8K dataset (paperwithcode, 2024). While constructing harder benchmarks is possible, their creation is often manual, expensive, and unscalable. As such, we present Scheherazade, an automated approach to produce large quantities of challenging mathematical reasoning benchmarks by logically chaining a small starting set of problems. We propose two different chaining methods, forward chaining and backward chaining, which include randomized branching techniques to generate complex reasoning problems. We apply Scheherazade on GSM8K to create GSM8K-Scheherazade and evaluate 3 frontier LLMs and OpenAI's o1-preview on it. We show that while other frontier models' performance declines precipitously at only a few questions chained, our evaluation suggests o1-preview's performance persists, with the flagship OpenAI model the only one to perform better at backward reasoning. Our data and code are available at https://github.com/YoshikiTakashima/scheherazade-code-data.

Scheherazade: Evaluating Chain-of-Thought Math Reasoning in LLMs with Chain-of-Problems

TL;DR

The paper addresses the fragility of existing math benchmarks as LLMs approach saturation by proposing Scheherazade, an automated benchmark-generation framework that chains problems via forward and backward reasoning structures. By linking problems with conditional branches and randomness, Scheherazade creates large, nested reasoning tasks from a small seed set and renders scalable evaluation feasible. Evaluations across OpenAI o1, GPT-4o, Claude 3.5 Sonnet, and Llama 3.1 show pronounced declines in both raw and normalized accuracy as chain length increases, with o1-preview exhibiting relatively robust backward reasoning. Error analysis distinguishes semantic understanding failures, wrong-path decisions, false negatives, and rare infinite-loop behavior, providing actionable insights for designing more discriminative benchmarks and understanding model reasoning patterns.

Abstract

Benchmarks are critical for measuring Large Language Model (LLM) reasoning capabilities. Some benchmarks have even become the de facto indicator of such capabilities. However, as LLM reasoning capabilities improve, existing widely-used benchmarks such as GSM8K marginally encapsulate model reasoning differentials - most state-of-the-art models for example achieve over 94% accuracy on the GSM8K dataset (paperwithcode, 2024). While constructing harder benchmarks is possible, their creation is often manual, expensive, and unscalable. As such, we present Scheherazade, an automated approach to produce large quantities of challenging mathematical reasoning benchmarks by logically chaining a small starting set of problems. We propose two different chaining methods, forward chaining and backward chaining, which include randomized branching techniques to generate complex reasoning problems. We apply Scheherazade on GSM8K to create GSM8K-Scheherazade and evaluate 3 frontier LLMs and OpenAI's o1-preview on it. We show that while other frontier models' performance declines precipitously at only a few questions chained, our evaluation suggests o1-preview's performance persists, with the flagship OpenAI model the only one to perform better at backward reasoning. Our data and code are available at https://github.com/YoshikiTakashima/scheherazade-code-data.
Paper Structure (26 sections, 9 equations, 3 figures, 1 table)

This paper contains 26 sections, 9 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Chain of Thought Prompting
  4. Synthetic math reasoning benchmark creation
  5. Approach
  6. Forward Chaining
  7. Backward Chaining
  8. Generation Power
  9. Evaluation
  10. Raw Accuracy
  11. Normalized Accuracy
  12. Error Analysis
  13. Semantic Understanding Errors
  14. Wrong Path Errors
  15. False Negatives
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Forward chaining generalization and example.
  • Figure 2: Accuracy of LLMs declines when the chains become longer. With the exception of o1-preview, LLMs find backward chains harder than forward chains at longer lengths.
  • Figure 3: Pie charts showing the error categorization of errors made by o1-preview and GPT-4o, both for forward and backward chaining.