FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees

TL;DR

FactTest is introduced, a novel framework that statistically assesses whether a LLM can confidently provide correct answers to given questions with high-probability correctness guarantees and ensures strong Type II error control under mild conditions and can be extended to maintain its effectiveness when covariate shifts exist.

Abstract

The propensity of Large Language Models (LLMs) to generate hallucinations and non-factual content undermines their reliability in high-stakes domains, where rigorous control over Type I errors (the conditional probability of incorrectly classifying hallucinations as truthful content) is essential. Despite its importance, formal verification of LLM factuality with such guarantees remains largely unexplored. In this paper, we introduce FactTest, a novel framework that statistically assesses whether a LLM can confidently provide correct answers to given questions with high-probability correctness guarantees. We formulate factuality testing as hypothesis testing problem to enforce an upper bound of Type I errors at user-specified significance levels. Notably, we prove that our framework also ensures strong Type II error control under mild conditions and can be extended to maintain its effectiveness when covariate shifts exist. Our approach is distribution-free and works for any number of human-annotated samples. It is model-agnostic and applies to any black-box or white-box LM. Extensive experiments on question-answering (QA) and multiple-choice benchmarks demonstrate that FactTest effectively detects hallucinations and improves the model's ability to abstain from answering unknown questions, leading to an over 40% accuracy improvement.

Figures (9)

• Figure 1: FactTest can control the Type I error given a significance level $\alpha$. The caption of each sub-figure consists of the dataset name and the model size.
• Figure 2: The Accuracy-Threshold curve. The title of each sub-figure consists of the dataset name, the model size and the certainty function.
• Figure 3: The Accuracy performance (%) of FactTest trained on half of the data, comparing with training-based baselines. Both R-Tuning and Finetune All utilize all training data for finetuning, while Finetune Half uses the same half of the finetuning data as FactTest.
• Figure 4: (a) The Accuracy performance (%) of FactTest on ParaRel-OOD testing dataset. (b)(c) FactTestO maintains its ability to control Type I error given a significance level $\alpha$ when distribution shifts exist.
• Figure 5: The certain and uncertain data distribution of the originated datasets obtained from supervised identification strategy. The title of each sub-figure consists of the dataset name and the size of the pre-trained model used to evaluate.

Theorems & Definitions (7)

• Theorem 1
• Theorem 2
• Theorem 3
• Proof 1: Proof of Theorem \ref{['thm:type1']}
• Lemma 1: Theorem 1 in skorski2023bernstein
• Proof 2: Proof of Theorem \ref{['thm:power']}
• Proof 3: Proof of Theorem \ref{['thm:type1_shift']}