Multi-LLM Adaptive Conformal Inference for Reliable LLM Responses

TL;DR

This work addresses the challenge of guaranteeing factuality in LLM outputs for high-stakes domains by reformulating conformal inference as a multiplicative filtering framework where the document-level factuality is modeled as the product of per-claim scores, enabling a group-conditional, finite-sample guarantee at user-specified level $1-\alpha$. It introduces MACI, which combines a Multi-LLM ensemble to produce higher-quality factuality-scores with group-specific calibration, yielding exact group-conditional coverage and superior retention compared to baselines. Theoretical results connect the quality of the factuality-score to retention via a margin-based bound, and empirical results on MedLFQA, WikiBio, and ExpertQA demonstrate stronger retention at target coverage and improved efficiency, including under covariate shift with MACI-DRE. The approach offers a practical, distribution-free solution for deploying LLMs in medicine and law, balancing reliability with information preservation for high-throughput applications.

Abstract

Ensuring factuality is essential for the safe use of Large Language Models (LLMs) in high-stakes domains such as medicine and law. Conformal inference provides distribution-free guarantees, but existing approaches are either overly conservative, discarding many true-claims, or rely on adaptive error rates and simple linear models that fail to capture complex group structures. To address these challenges, we reformulate conformal inference in a multiplicative filtering setting, modeling factuality as a product of claim-level scores. Our method, Multi-LLM Adaptive Conformal Inference (MACI), leverages ensembles to produce more accurate factuality-scores, which in our experiments led to higher retention, while validity is preserved through group-conditional calibration. Experiments show that MACI consistently achieves user-specified coverage with substantially higher retention and lower time cost than baselines. Our repository is available at https://github.com/MLAI-Yonsei/MACI

Figures (7)

• Figure 1: Comparison of Conformal Inference Methods. T (true) and F (false) denote ground-truth labels per claim. Basic Conformal Inference mohri2024languagemodelsconformalfactuality attains coverage by aggressive filtering, yielding low retention. Conditional Conformal Inference cherian2024largelanguagemodelvalidity proposes adaptive thresholds but relaxes guarantees; MACI achieves both high coverage and retention.
• Figure 2: Performance comparison of CCI (adaptive $\alpha$) and MACI measured by View Count on the WikiBio dataset. The horizontal axis of the left graph is the sample index sorted by View Count, and the vertical axis is $\alpha$. The left graph shows the variation in $\alpha$ when CCI (adaptive $\alpha$) sets its target retention ratio to MACI's average retention ratio. CCI (adaptive $\alpha$) trades off higher $\alpha$ to achieve a higher retention ratio, and the table below shows the resulting decrease in coverage.
• Figure 3: (a) shows the high Jaccard distance between different LLMs’ predictions on claims known to be false in MedLFQA, indicating diverse false-claim detection patterns that support using an ensemble. (b) demonstrates the sequential improvement in FPR from a single-LLM and a simple arithmetic mean ensemble to our proposed MACI. It also demonstrates that as the FPR improves, the MSE also improves in practice; (c) demonstrates that as the FPR and MSE improve, the retention ratio also increases. A more detailed analysis is in Figure \ref{['fig:Ensemble_full_analysis']}.
• Figure 4: An example of independently decomposed claims in MedLFQA and the aggregated results of four methods that filter the false-claims of those claims. BCI yields conservative results, while CCI and FSC-KG show high retention but fail to filter out all false-claims, whereas MACI successfully filters out all false-claims.
• Figure 5: Coverage and retention in large number of groups, using MACI and the MACI-cluster method. The top shows results split into 24 groups. The average coverage per group is more maintained by MACI than MACI-cluster, but with greater variance. Using the group clustering method allows for practical results by sacrificing strict group-conditional coverage.

Theorems & Definitions (8)

• Definition 1: Oracle Filtering Rule
• Theorem 1: Marginal Coverage Guarantee
• Theorem 2: Group-conditional Coverage Guarantee
• Theorem 3: Retention gap with MSE
• Lemma 1
• proof
• Lemma 2: Uniformity under oracle factuality-score
• proof