Large Language Models Help Humans Verify Truthfulness -- Except When They Are Convincingly Wrong

TL;DR

The paper evaluates whether LLM-generated natural language explanations aid humans in fact-checking claims as effectively as retrieval of supporting passages. Through a large-scale human study on FoolMeTwice data, it compares Baseline, Retrieval, Explanation, Contrastive Explanation, and Retrieval+Explanation conditions, finding that explanations speed verification with similar accuracy to retrieval but can cause over-reliance when wrong. Contrastive explanations mitigate some over-trust but reduce accuracy when non-contrastive explanations are correct, and combining retrieval with explanations yields no clear benefit. The results suggest NL explanations are not a reliable substitute for retrieved evidence in high-stakes fact-checking and highlight the need for more adaptive or calibrated explanations. The work raises important implications for human-AI collaboration in information verification and misinformation mitigation.

Abstract

Large Language Models (LLMs) are increasingly used for accessing information on the web. Their truthfulness and factuality are thus of great interest. To help users make the right decisions about the information they get, LLMs should not only provide information but also help users fact-check it. Our experiments with 80 crowdworkers compare language models with search engines (information retrieval systems) at facilitating fact-checking. We prompt LLMs to validate a given claim and provide corresponding explanations. Users reading LLM explanations are significantly more efficient than those using search engines while achieving similar accuracy. However, they over-rely on the LLMs when the explanation is wrong. To reduce over-reliance on LLMs, we ask LLMs to provide contrastive information - explain both why the claim is true and false, and then we present both sides of the explanation to users. This contrastive explanation mitigates users' over-reliance on LLMs, but cannot significantly outperform search engines. Further, showing both search engine results and LLM explanations offers no complementary benefits compared to search engines alone. Taken together, our study highlights that natural language explanations by LLMs may not be a reliable replacement for reading the retrieved passages, especially in high-stakes settings where over-relying on wrong AI explanations could lead to critical consequences.

Figures (9)

• Figure 1: An example claim and the corresponding ChatGPT explanation, retrieved passages (abridged), and contrastive explanation. The claim is true and the refuting explanation has factual errors and reasoning contradiction.
• Figure 2: Human decision accuracy and average time spent on verifying a claim. Both retrieval and explanation significantly improve human verification accuracy, while explanation takes a significantly shorter time.
• Figure 3: Human verification accuracy broken down into two subsets: examples on which the explanation gives the correct labels, and examples on which the explanation gives the wrong labels. Humans over-rely on explanations so that they achieve significantly lower accuracy than the baseline when the explanation is wrong.
• Figure 4: Verification accuracy and time broken down by whether the (non-contrastive) explanation is correct. Contrastive explanation significantly improves accuracy over non-contrastive explanation on examples where the non-contrastive explanation is wrong, with some drop in accuracy on examples where the non-contrastive explanation is correct.
• Figure 5: Verification accuracy and time breakdown. Combining retrieval and explanation is not significantly better than just showing retrieved passages alone.