Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities

Tianyi Zhuang, Chuqiao Kuang, Xiaoguang Li, Yihua Teng, Jihao Wu, Yasheng Wang, Lifeng Shang

TL;DR

DocPuzzle introduces a rigorous, process-aware benchmark for evaluating long-context reasoning in LLMs using 100 expert-annotated, multi-domain QA tasks grounded in long real-world documents. A human–AI collaboration and a checklist-based evaluation decouple reasoning validity from final answers, addressing guessing biases and enabling detailed analysis of intermediate reasoning. Experiments show slow-thinking models outperform typical instruct models, while distilled reasoning transfers poorly to complex long-context tasks, underscoring limits of distillation for advanced reasoning. The benchmark, with its multi-domain scope and validation pipeline, offers a robust framework for benchmarking reasoning processes and guiding future research toward more generalizable, reasoning-rich AI systems.

Abstract

We present DocPuzzle, a rigorously constructed benchmark for evaluating long-context reasoning capabilities in large language models (LLMs). This benchmark comprises 100 expert-level QA problems requiring multi-step reasoning over long real-world documents. To ensure the task quality and complexity, we implement a human-AI collaborative annotation-validation pipeline. DocPuzzle introduces an innovative evaluation framework that mitigates guessing bias through checklist-guided process analysis, establishing new standards for assessing reasoning capacities in LLMs. Our evaluation results show that: 1)Advanced slow-thinking reasoning models like o1-preview(69.7%) and DeepSeek-R1(66.3%) significantly outperform best general instruct models like Claude 3.5 Sonnet(57.7%); 2)Distilled reasoning models like DeepSeek-R1-Distill-Qwen-32B(41.3%) falls far behind the teacher model, suggesting challenges to maintain the generalization of reasoning capabilities relying solely on distillation.

DocPuzzle: A Process-Aware Benchmark for Evaluating Realistic Long-Context Reasoning Capabilities

TL;DR

DocPuzzle introduces a rigorous, process-aware benchmark for evaluating long-context reasoning in LLMs using 100 expert-annotated, multi-domain QA tasks grounded in long real-world documents. A human–AI collaboration and a checklist-based evaluation decouple reasoning validity from final answers, addressing guessing biases and enabling detailed analysis of intermediate reasoning. Experiments show slow-thinking models outperform typical instruct models, while distilled reasoning transfers poorly to complex long-context tasks, underscoring limits of distillation for advanced reasoning. The benchmark, with its multi-domain scope and validation pipeline, offers a robust framework for benchmarking reasoning processes and guiding future research toward more generalizable, reasoning-rich AI systems.

Abstract

We present DocPuzzle, a rigorously constructed benchmark for evaluating long-context reasoning capabilities in large language models (LLMs). This benchmark comprises 100 expert-level QA problems requiring multi-step reasoning over long real-world documents. To ensure the task quality and complexity, we implement a human-AI collaborative annotation-validation pipeline. DocPuzzle introduces an innovative evaluation framework that mitigates guessing bias through checklist-guided process analysis, establishing new standards for assessing reasoning capacities in LLMs. Our evaluation results show that: 1)Advanced slow-thinking reasoning models like o1-preview(69.7%) and DeepSeek-R1(66.3%) significantly outperform best general instruct models like Claude 3.5 Sonnet(57.7%); 2)Distilled reasoning models like DeepSeek-R1-Distill-Qwen-32B(41.3%) falls far behind the teacher model, suggesting challenges to maintain the generalization of reasoning capabilities relying solely on distillation.

Paper Structure

This paper contains 33 sections, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Context-free Reasoning Benchmarks
  4. Long-context Reasoning Benchmarks
  5. Elementary Long-context Benchmarks
  6. Difficult Long-context Reasoning Benchmarks
  7. DocPuzzle
  8. Data Construction Pipeline
  9. Data Collection
  10. Data Annotation
  11. Data Verification and Revision
  12. Evaluation Method
  13. Experiments
  14. Baselines
  15. Slow-Thinking Reasoning Models
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Overview of DocPuzzle construction pipeline. There are five stages before the samples are included in the benchmark.
  • Figure 2: Distribution of document categories in DocPuzzle benchmark.
  • Figure 3: Response length vs. accuracy relationship across model types.
  • Figure 4: The performance of DeepSeek-R1 and DeepSeek-R1-Distill-Qwen-32B on different benchmarks.
  • Figure 5: Measuring exploration potential via pass@3 metric.