Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?

Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Zhongyuan Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, Bo Zheng

TL;DR

DeltaBench is a comprehensive benchmark for evaluating the quality of long Chain-of-Thought reasoning generated by o1-like LLMs and the critique capabilities of existing PRMs and LLMs. It introduces a 1,236-sample dataset spanning Math, Programming, PCB, and General Reasoning with per-section annotations to locate errors, reflections, and usefulness, plus an outlier-based scoring framework. Key findings reveal prevalent fundamental and reasoning errors in long CoTs, limited effectiveness of current PRMs and self-critique, and that GPT-4-turbo-128k remains the strongest critic across tasks, while o1-like approaches do not consistently outperform baselines. The work provides actionable insights for improving long CoT generation, error detection, and critique mechanisms, and establishes a rigorous, section-level evaluation protocol to guide future research and development.

Abstract

Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.

Can Large Language Models Detect Errors in Long Chain-of-Thought Reasoning?

TL;DR

DeltaBench is a comprehensive benchmark for evaluating the quality of long Chain-of-Thought reasoning generated by o1-like LLMs and the critique capabilities of existing PRMs and LLMs. It introduces a 1,236-sample dataset spanning Math, Programming, PCB, and General Reasoning with per-section annotations to locate errors, reflections, and usefulness, plus an outlier-based scoring framework. Key findings reveal prevalent fundamental and reasoning errors in long CoTs, limited effectiveness of current PRMs and self-critique, and that GPT-4-turbo-128k remains the strongest critic across tasks, while o1-like approaches do not consistently outperform baselines. The work provides actionable insights for improving long CoT generation, error detection, and critique mechanisms, and establishes a rigorous, section-level evaluation protocol to guide future research and development.

Abstract

Recently, o1-like models have drawn significant attention, where these models produce the long Chain-of-Thought (CoT) reasoning steps to improve the reasoning abilities of existing Large Language Models (LLMs). In this paper, to understand the qualities of these long CoTs and measure the critique abilities of existing LLMs on these long CoTs, we introduce the DeltaBench, including the generated long CoTs from different o1-like models (e.g., QwQ, DeepSeek-R1) for different reasoning tasks (e.g., Math, Code, General Reasoning), to measure the ability to detect errors in long CoT reasoning. Based on DeltaBench, we first perform fine-grained analysis of the generated long CoTs to discover the effectiveness and efficiency of different o1-like models. Then, we conduct extensive evaluations of existing process reward models (PRMs) and critic models to detect the errors of each annotated process, which aims to investigate the boundaries and limitations of existing PRMs and critic models. Finally, we hope that DeltaBench could guide developers to better understand the long CoT reasoning abilities of their models.

Paper Structure

This paper contains 53 sections, 1 equation, 16 figures, 8 tables.

Table of Contents

  1. Introduction
  2. DeltaBench
  3. Dataset Construction
  4. Query collection.
  5. Data Preprocessing.
  6. Long CoT Generation.
  7. Section Split.
  8. Correctness Assessment
  9. Human Annotation
  10. Dataset Statistics
  11. Evaluation Metrics
  12. Comparison to Other Benchmarks
  13. Analysis
  14. Error Analysis of o1-like Models
  15. Error Type Lists
  16. ...and 38 more sections

Figures (16)

  • Figure 1: Illustration of the evaluation process for critic models and Process Reward Models (PRMs) for DeltaBench.
  • Figure 2: Left: Overview of DeltaBench. These pie charts show the distribution of questions in Math, General Reasoning, PCB (Physics, Chemistry and Biology), and Programming. Right: Statistics of DeltaBench.
  • Figure 3: An example of section division for long CoT reasoning process.
  • Figure 4: An example of human annotation applied to a mathematical problem-solving process. Annotators are required to annotate each section individually.
  • Figure 5: Distribution of long CoT characteristics.
  • ...and 11 more figures