Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertainty

TL;DR

The paper addresses the challenge of estimating true uncertainty in black-box LLMs, where self-consistency methods can mislead due to contextual biases in knowledge retrieval. It introduces DiverseAgentEntropy, a same-model, multi-agent framework that uses diverse, knowledge-preserving perturbations of a target query to collaboratively refine responses and obtain calibrated uncertainty estimates. Empirical results show improved AUROC for uncertainty and better hallucination detection compared with self-consistency baselines across several benchmarks and two models, albeit with higher computational cost. The work highlights the importance of diversifying queries and leveraging intra-model collaboration to better surface the model's true knowledge and uncertainty, with implications for scalable oversight in high-stakes applications.

Abstract

Quantifying uncertainty in black-box LLMs is vital for reliable responses and scalable oversight. Existing methods, which gauge a model's uncertainty through evaluating self-consistency in responses to the target query, can be misleading: an LLM may confidently provide an incorrect answer to a target query, yet give a confident and accurate answer to that same target query when answering a knowledge-preserving perturbation of the query. We systematically analyze the model behaviors and demonstrate that this discrepancy stems from suboptimal retrieval of parametric knowledge, often due to contextual biases that prevent consistent access to stored knowledge. We then introduce DiverseAgentEntropy, a novel, theoretically-grounded method employing multi-agent interaction across diverse query variations for uncertainty estimation of black-box LLMs. This approach more accurately assesses an LLM's true uncertainty and improves hallucination detection, outperforming existing self-consistency based techniques.

Figures (21)

• Figure 1: Black-box methods relying on self-consistency (pink box) misestimate model uncertainty due to a mismatch between the uncertainty estimated from the original target query and the model’s actual knowledge, while DiverseAgentEntropy (blue box) recovers true model uncertainty.
• Figure 2: DiverseAgentEntropy estimates model uncertainty by enabling multi-agent interactions on diverse knowledge-perserving queries, analyzing uncertainty based on these interactions rather than simple self-consistency.
• Figure 3: AR-curves across all data. SC refers to SC (SE). SC w 5 questions refers to calculating entropy using the agents' diverse questions without agent interaction. We present the best and sub optimal methods for each method category in Tables \ref{['tab:auroc']} and \ref{['tab:main']}.
• Figure 4: KL-Divergence with regard to interaction rounds for three examples where the model knows the answer.
• Figure 5: Effect of $\#$agents on performance.

Theorems & Definitions (4)

• Definition 4.1: Target knowledge Derivation
• Definition 4.2: Knowledge-Preserving Perturbation of the Target Query
• Definition 4.3: Induced Distribution for Target Query $q$
• Theorem 4.9: Convergence to True Distribution When Known