Can We Verify Step by Step for Incorrect Answer Detection?

TL;DR

This work tackles whether final answer correctness can be predicted by inspecting multi-chain reasoning in LLMs. It introduces R2PE, a comprehensive benchmark spanning eight datasets, five domains, and six LLMs to study the link between rationales and performance, and proposes the Process Discernibility Score (PDS) to aggregate rationale-level information for verification. Empirically, PDS consistently outperforms the previous answer-consensus baseline (ADS) in F1 and AUC-PR across 45 subsets and can further boost downstream open-domain QA performance when combined with verify-and-edit. These findings demonstrate the value of process-aware verification in chain-of-thought reasoning and suggest practical gains for improving LLM reliability in real-world tasks.

Abstract

Chain-of-Thought (CoT) prompting has marked a significant advancement in enhancing the reasoning capabilities of large language models (LLMs). Previous studies have developed various extensions of CoT, which focus primarily on enhancing end-task performance. In addition, there has been research on assessing the quality of reasoning chains in CoT. This raises an intriguing question: Is it possible to predict the accuracy of LLM outputs by scrutinizing the reasoning chains they generate? To answer this research question, we introduce a benchmark, R2PE, designed specifically to explore the relationship between reasoning chains and performance in various reasoning tasks spanning five different domains. This benchmark aims to measure the falsehood of the final output of LLMs based on the reasoning steps. To make full use of information in multiple reasoning chains, we propose the process discernibility score (PDS) framework that beats the answer-checking baseline by a large margin. Concretely, this resulted in an average of $5.1\%$ increase in the F1 score and $2.97\%$ improvement in AUC-PR across all 45 subsets within R2PE. We further demonstrate our PDS's efficacy in advancing open-domain QA accuracy.

Figures (5)

• Figure 1: An example from HotpotQA that GPT-4 outputs highly consistent but wrong answers. In this particular example, our PDS can detect conflicting information about the actor (colored by red) and predicts that the answer will be incorrect because it is less than zero, while ADS predicts the answer to be correct because it is greater than 2.5.
• Figure 2: An overview of R2PE benchmark and PDS framework. The construction of R2PE includes 5 stages: data collection, response generation, extraction, manual cleaning, and labeling. Sourced from 8 distinct datasets covering a variety of task types and derived from 6 different LLMs, R2PE comprises 45 subsets, featuring approximately 20.2K TRUE and 18.1K FALSE instances across 5 domains. The objective is to establish a discernibility score that accurately indicates the veracity of answers. PDS adopts answer checking and process supervision to detect all potential discrepancies among different rationales, which beats ADS that focuses merely on the answer consensus.
• Figure 3: Results on the subset (HotpotQA from GPT-4): ADS (left), PDS (middle), and PDS-ADS (right). ADS has high precision: large answer agreement does not always guarantee accurate predictions. PDS yields desirable outcomes, while PDS-ADS has poor precision.
• Figure 4: PDS can be integrated with verify-and-edit to further improve accuracy on open-domain QA.
• Figure 5: ROC Curve of PDS on (HotpotQA from GPT3).