Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models

TL;DR

The DetectBench is introduced, a reading comprehension dataset designed to assess a model's ability to jointly ability in key information detection and multi-hop reasoning when facing complex and implicit information.

Abstract

Detectives frequently engage in information detection and reasoning simultaneously when making decisions across various cases, especially when confronted with a vast amount of information. With the rapid development of large language models~(LLMs), evaluating how these models identify key information and reason to solve questions becomes increasingly relevant. We introduces the DetectBench, a reading comprehension dataset designed to assess a model's ability to jointly ability in key information detection and multi-hop reasoning when facing complex and implicit information. The DetectBench comprises 3,928 questions, each paired with a paragraph averaging 190 tokens in length. To enhance model's detective skills, we propose the Detective Thinking Framework. These methods encourage models to identify all possible clues within the context before reasoning. Our experiments reveal that existing models perform poorly in both information detection and multi-hop reasoning. However, the Detective Thinking Framework approach alleviates this issue.

Figures (6)

• Figure 1: When facing overloaded information LLMs may produce outputs arbitrarily due to their inability to engage in deep contemplation. In contrast, humans who are experienced, like detectives, analyze and correlate all available information, thereby identifying pivotal clues that lead to the answer of the problem.
• Figure 2: The example of the question in DetectBench
• Figure 3: Within the Detective Thinking Prompt paradigm, the process is bifurcated into distinct phases: Detail Detection and Detail Connection, followed by Answer Inspiration and Weighted Reasoning. The Detective Thinking Finetune strategy is predominantly aimed at collecting data for fine-tuning. The first three phases permits free generation via open-source models, culminating in the aggregation of these outputs into a cohesive answer during the final stage.
• Figure 4: The Pearson Correlation between the KeyInfo. metric and the Accuracy metric across all models and prompt methods.
• Figure 5: The performance of GPT4-Turbo is correlated with the context length, option length, the quantity of clues, and the number of reasoning steps involved.