Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?

Zhouhong Gu, Lin Zhang, Xiaoxuan Zhu, Jiangjie Chen, Wenhao Huang, Yikai Zhang, Shusen Wang, Zheyu Ye, Yan Gao, Hongwei Feng, Yanghua Xiao

TL;DR

A benchmark called DetectBench is proposed for verifying the ability to detect and piece together implicit evidence within a long context and the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs.

Abstract

Detecting evidence within the context is a key step in the process of reasoning task. Evaluating and enhancing the capabilities of LLMs in evidence detection will strengthen context-based reasoning performance. This paper proposes a benchmark called DetectBench for verifying the ability to detect and piece together implicit evidence within a long context. DetectBench contains 3,928 multiple-choice questions, with an average of 994 tokens per question. Each question contains an average of 4.55 pieces of implicit evidence, and solving the problem typically requires 7.62 logical jumps to find the correct answer. To enhance the performance of LLMs in evidence detection, this paper proposes Detective Reasoning Prompt and Finetune. Experiments demonstrate that the existing LLMs' abilities to detect evidence in long contexts are far inferior to humans. However, the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs. Moreover, when the abilities of LLMs in evidence detection are improved, their final reasoning performance is also enhanced accordingly.

DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?

TL;DR

A benchmark called DetectBench is proposed for verifying the ability to detect and piece together implicit evidence within a long context and the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs.

Abstract

Detecting evidence within the context is a key step in the process of reasoning task. Evaluating and enhancing the capabilities of LLMs in evidence detection will strengthen context-based reasoning performance. This paper proposes a benchmark called DetectBench for verifying the ability to detect and piece together implicit evidence within a long context. DetectBench contains 3,928 multiple-choice questions, with an average of 994 tokens per question. Each question contains an average of 4.55 pieces of implicit evidence, and solving the problem typically requires 7.62 logical jumps to find the correct answer. To enhance the performance of LLMs in evidence detection, this paper proposes Detective Reasoning Prompt and Finetune. Experiments demonstrate that the existing LLMs' abilities to detect evidence in long contexts are far inferior to humans. However, the Detective Reasoning Prompt effectively enhances the capability of powerful LLMs in evidence detection, while the Finetuning method shows significant effects in enhancing the performance of weaker LLMs. Moreover, when the abilities of LLMs in evidence detection are improved, their final reasoning performance is also enhanced accordingly.
Paper Structure (29 sections, 6 figures, 22 tables)

This paper contains 29 sections, 6 figures, 22 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Information Retrieval
  4. Commonsense Reasoning
  5. Detective Benchmark
  6. Construction
  7. Statistic
  8. Human Performance
  9. Detective Reasoning
  10. Detective Reasoning Prompt
  11. Detective Reasoning Finetune
  12. Experiments
  13. Overall Setup
  14. Performance with Different Prompt
  15. Performance of Finetuning
  16. ...and 14 more sections

Figures (6)

  • Figure 1: LLMs are hard to aware of the implicit evidence in the context so they may respond arbitrarily.
  • Figure 2: The example of the question in DetectBench.
  • Figure 3: The figure represents the conceptual framework of "Detective Reasoning".
  • Figure 4: The Pearson Correlation between the evidence detection (RougeL) and reasoning performance (Accuracy) across all models and prompt methods.
  • Figure 5: The performance of GPT4-Turbo is correlated with the context length, option length, the number of evidence, and the number of reasoning steps involved.
  • ...and 1 more figures