Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal

TL;DR

This work tackles the problem of trustworthy uncertainty in large language models by introducing UniCR, a framework that fuses heterogeneous uncertainty signals into a calibrated probability of correctness and enforces a user-defined error budget through principled refusal. It integrates sequence likelihood, self-consistency, retrieval compatibility, and verifier signals, and learns a lightweight calibration head with temperature scaling; conformal risk control provides distribution-free guarantees on coverage under a specified risk. UniCR demonstrates improved calibration and safer refusal across short-form QA, code generation, and long-form retrieval-based QA, with robust performance under distribution shift and API-only (logit-free) settings. The approach yields informative user-facing refusal messages, is portable across backends and tasks, and aligns confidence with semantic factuality to reduce hallucinations while preserving coverage. Overall, UniCR offers a practical, theoretically grounded recipe for trusted uncertainty in AI systems without requiring fine-tuning of the base model.

Abstract

Deployed language models must decide not only what to answer but also when not to answer. We present UniCR, a unified framework that turns heterogeneous uncertainty evidence including sequence likelihoods, self-consistency dispersion, retrieval compatibility, and tool or verifier feedback into a calibrated probability of correctness and then enforces a user-specified error budget via principled refusal. UniCR learns a lightweight calibration head with temperature scaling and proper scoring, supports API-only models through black-box features, and offers distribution-free guarantees using conformal risk control. For long-form generation, we align confidence with semantic fidelity by supervising on atomic factuality scores derived from retrieved evidence, reducing confident hallucinations while preserving coverage. Experiments on short-form QA, code generation with execution tests, and retrieval-augmented long-form QA show consistent improvements in calibration metrics, lower area under the risk-coverage curve, and higher coverage at fixed risk compared to entropy or logit thresholds, post-hoc calibrators, and end-to-end selective baselines. Analyses reveal that evidence contradiction, semantic dispersion, and tool inconsistency are the dominant drivers of abstention, yielding informative user-facing refusal messages. The result is a portable recipe of evidence fusion to calibrated probability to risk-controlled decision that improves trustworthiness without fine-tuning the base model and remains valid under distribution shift.

Figures (6)

• Figure 1: System overview of UniCR: multi-source evidence is fused into a calibrated correctness probability and thresholded to produce answer-or-refusal under a risk budget.
• Figure 2: Reliability diagram of calibrated confidence $c(x)$. The closer to the diagonal, the better.
• Figure 3: Risk--coverage (RC) curves comparing UniCR with common baselines (lower is better).
• Figure 4: Ablation on evidence families and control strategies, reported as AURC (lower is better).
• Figure 5: Relative latency vs. self-consistency samples $K$.