Conformal Language Model Reasoning with Coherent Factuality

TL;DR

This work tackles the challenge of ensuring factual and coherent reasoning in language model outputs. It introduces coherent factuality, a notion that enforces both accuracy and proper substantiation across a reasoning chain using deducibility graphs, and couples it with a subgraph-focused split conformal prediction protocol to guarantee user-specified coverage. By constructing ideal or approximate deducibility graphs and applying graph-aware scoring, the approach achieves high factuality while retaining a large fraction of the original claims on math and verbal reasoning benchmarks like $MATH$ and $FELM$. The empirical results demonstrate calibrated coherent factuality, the necessity of graph proxies for coherence, and benefits from bootstrapping coherent inputs, illustrating a viable path toward more reliable reasoning in large language models. The methodology has potential implications for a range of reasoning-intensive tasks and domains, including code generation and beyond.

Abstract

Language models are increasingly being used in important decision pipelines, so ensuring the correctness of their outputs is crucial. Recent work has proposed evaluating the "factuality" of claims decomposed from a language model generation and applying conformal prediction techniques to filter out those claims that are not factual. This can be effective for tasks such as information retrieval, where constituent claims may be evaluated in isolation for factuality, but is not appropriate for reasoning tasks, as steps of a logical argument can be evaluated for correctness only within the context of the claims that precede them. To capture this, we define "coherent factuality" and develop a conformal-prediction-based method to guarantee coherent factuality for language model outputs. Our approach applies split conformal prediction to subgraphs within a "deducibility" graph" that represents the steps of a reasoning problem. We evaluate our method on mathematical reasoning problems from the MATH and FELM datasets and find that our algorithm consistently produces correct and substantiated orderings of claims, achieving coherent factuality across target coverage levels. Moreover, we achieve 90% factuality on our stricter definition while retaining 80% or more of the original claims, highlighting the utility of our deducibility-graph-guided approach.

Figures (8)

• Figure 1: Here, the previous method (Output 1) removes the erroneous claims outlined in red, but leaves the response incoherent by removing Step 2, which is referenced in Step 3. We (Output 2) consider reasoning structure to filter out erroneous claims while maintaining coherence; even though we remove a true claim, it is not essential for understanding the claims that remain ($\alpha = 0.1$).
• Figure 2: The nodes above correspond to the subclaims enumerated in Figure \ref{['fig:prelim']}. In blue is the ideal deducibility graph for this output which gives perfect information and allows us to keep all true claims. Even though our approximate deducibility graph lacks a ground truth node and has additional edges (e.g., $(6,7)$), it helps us preserve the integrity of an output while filtering. In contrast, the baseline method leaves Claim 3 unsubstantiated by omitting Claim 2.
• Figure 3: Even though Claim 6 is technically true, it detracts from the coherent solution as it is derived from a false claim (which suggests the solution is 0). Although we do not require dependency, the edge $(5,6)$ prevents consideration of Claim 6 in the absence of Claim 5. This property improves the quality of the subgraphs we consider.
• Figure 4: We evaluate our post-hoc (green) and subgraph filtering algorithms (using descendant weighting with $\beta = 1/2$ (red) and graph-independent scoring (blue)) on MATH dataset. Post-hoc filtering is applied using the graph after initial filtering without a graph. We consider the baseline to be the method of mohri2024languagemodelsconformalfactuality (yellow). In (a), we show calibration to desired factuality levels for Subgraph Filtering within theoretical bounds (shown in grey). In (b), we assess claim retention rates by varying $\alpha$ values, plotting both realized factuality and the fraction of retained claims across calibration methods and graph generation techniques. In (c) we plot claim retention with respect to user-desired calibration level.
• Figure 5: Results on the FELM dataset using GPT-4 for responses and GPT-4o for graphs.

Theorems & Definitions (15)

• Definition 1: Claim
• Definition 2: Ground truth
• Remark 1
• Definition 3: Coherent factuality
• Remark 2
• Remark 3
• Definition 4: Approximate deducibility graph
• Remark 4
• Definition 5: Non-conformity scoring function
• Theorem 1: Calibrated Factuality