Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals

TL;DR

This paper addresses the challenge of reliable confidence estimation for large language models under distribution shift by introducing Structural Confidence, a single-pass, model-agnostic framework that leverages multi-scale structural signals from a proxy hidden-state trajectory. The method extracts spectral stability, local variation, and shape coherence descriptors, concatenates them into a fixed-size representation, and uses a lightweight predictor to estimate correctness, optionally augmenting with a semantic embedding feature. Across FEVER, SciFact, WikiBio, and TruthfulQA, the approach achieves strong AUROC and AUPR while significantly reducing compute relative to sampling-based baselines, and shows robust cross-domain performance. The work demonstrates that internal dynamics of hidden-state trajectories provide a practical, robust modality for post-hoc confidence estimation in resource-constrained LLM applications, with potential extensions to open-weight and multimodal models.

Abstract

Large language models (LLMs) are increasingly deployed in domains where errors carry high social, scientific, or safety costs. Yet standard confidence estimators, such as token likelihood, semantic similarity and multi-sample consistency, remain brittle under distribution shift, domain-specialised text, and compute limits. In this work, we present Structural Confidence, a single-pass, model-agnostic framework that enhances output correctness prediction based on multi-scale structural signals derived from a model's final-layer hidden-state trajectory. By combining spectral, local-variation, and global shape descriptors, our method captures internal stability patterns that are missed by probabilities and sentence embeddings. We conduct extensive, cross-domain evaluation across four heterogeneous benchmarks-FEVER (fact verification), SciFact (scientific claims), WikiBio-hallucination (biographical consistency), and TruthfulQA (truthfulness-oriented QA). Our Structural Confidence framework demonstrates strong performance compared with established baselines in terms of AUROC and AUPR. More importantly, unlike sampling-based consistency methods which require multiple stochastic generations and an auxiliary model, our approach uses a single deterministic forward pass, offering a practical basis for efficient, robust post-hoc confidence estimation in socially impactful, resource-constrained LLM applications.

Figures (4)

• Figure 1: Overall Structural Confidence pipeline. An LLM produces a single deterministic answer; an encoder maps the (context, answer) to a hidden-state trajectory from which multi-scale structural descriptors are extracted and scored by a lightweight confidence model.
• Figure 2: Cross-domain AUROC for Structure-feature and Semantic-feature trained on mix_train and evaluated on four domains.
• Figure 3: AUROC of fft_only, lap_only, tda_only, geo_only, struct_only, sent_only, and struct_plus_sent across FEVER, SciFact, and WikiBio. Unified structural descriptors and the fused Struct+Sent variant show the most stable behaviour across datasets.
• Figure 4: Per-metric heatmap (AUROC and AUPR) for structural and semantic variants on SciFact. Semantic confidence exhibits a pronounced collapse relative to FEVER and WikiBio, while structural trajectories provide weaker but more stable signals across metrics.