Teaming LLMs to Detect and Mitigate Hallucinations

TL;DR

This work tackles the persistent problem of LLM hallucinations by introducing consortium consistency, a black-box ensemble approach that combines outputs from diverse LLMs. It extends single-model consistency via semantic clustering, consortium voting, and consortium entropy to form multi-model ensembles that improve hallucination detection and mitigation while often reducing inference costs. Across 11 tasks and 15 models, consortium confidence and performance gains depend on the composition and strength of the participating models, with strong, similarly capable ensembles delivering the largest improvements. The method offers practical guidance for deploying multi-LLM systems in real-world settings and points to future directions such as weighted aggregation and retrieval-augmented strategies to further enhance robustness.

Abstract

Recent work has demonstrated state-of-the-art results in large language model (LLM) hallucination detection and mitigation through consistency-based approaches which involve aggregating multiple responses sampled from a single LLM for a given prompt. These approaches help offset limitations stemming from the imperfect data on which LLMs are trained, which includes biases and under-representation of information required at deployment time among other limitations which can lead to hallucinations. We show that extending these single-model consistency methods to combine responses from multiple LLMs with different training data, training schemes and model architectures can result in substantial further improvements in hallucination detection and mitigation capabilities beyond their single-model consistency counterparts. We evaluate this "consortium consistency" approach across many model teams from a pool of 15 LLMs and explore under what conditions it is beneficial to team together different LLMs in this manner. Further, we show that these performance improvements often come with reduced inference costs, offsetting a significant drawback with single-model consistency methods.

Figures (6)

• Figure 1: Illustration of consortium consistency. A given query is input to multiple LLMs, one or more responses are sampled from each model. Semantically-equivalent answers are clustered together, and the probability distribution of different answers is computed from these clustered samples. The distribution is used to calculate a final answer to the query, and an entropy score. Queries with higher entropy have less consistent responses and are hence more likely to contain hallucinations. Combining responses from multiple different LLMs reduces the likelihood of incorrectly assigning high confidence to hallucinated answers and allows consistently hallucinating models to be out-voted by other models.
• Figure 2: (a) A representative example showing consortium consistency improving on average across 11 test sets over single-model consistency applied to each of the constituent models, across a range of sample budgets per-query. (b) Consortium consistency dominates single-model consistency on the cost-performance frontier, achieving both higher performance and lower cost simultaneously. X-axes show mean API cost in dollars per query, which grows with increasing number of sampled responses per query.
• Figure 3: KDE plots showing performance of consortia vs hard baselines as a function of varying properties of constituent models. Plots are generated using 1000 consortia randomly selected from all $2^{15} - 15$ consortia that can be formed from the available models.
• Figure 4: (a) The entropy distributions show a clearer separation between correct and incorrect responses for the consortium in the low-entropy region, even though Mythomax achieves the highest AUROC overall. This suggests that the consortium is better calibrated in high-confidence cases. (b) While the consortium’s improved low-entropy separation slightly boosts performance at the left-most part of the ROC curve, the overall AUROC remains highest for Mythomax. (c) Precision-recall curves reveal a more substantial benefit: while mythomax dominates in the higher recall range, the consortium attains higher peak precision, allowing more flexibility to trade off recall for higher precision. This is a common trend (see Appendix \ref{['appendix:PR_curves']}).
• Figure 5: Consortium entropy typically attains higher peak precision values than semantic entropy with any of the individual constituent models (controlled for number of responses per query). Consortia typically allow more flexibility to trade-off recall for precision, even in cases where some of the individual constituent models have higher AUROC scores (see \ref{['sec:entropy_precision']} for more discussion). Each sub-plot shows the precision-recall curve for a randomly chosen consortium (in blue) along with precision-recall curves for each of the constituent models. Consortia are chosen at random without filtering for variance in ability or mean ability, however they are filtered to consortia comprised of 4 models or less to aid readability.