Confidence-Aware Routing for Large Language Model Reliability Enhancement: A Multi-Signal Approach to Pre-Generation Hallucination Mitigation

TL;DR

This work tackles hallucination in large language models by shifting from post-generation corrections to pre-generation reliability assessment. It introduces a multi-signal confidence estimator that fuses semantic alignment, internal convergence, and learned uncertainty into a unified score $C_{\text{overall}}$, which deterministically routes queries to local generation, retrieval-augmented generation, larger models, or human review via $A(\cdot)$. Empirical results on knowledge-intensive QA benchmarks show improved hallucination detection (0.74) and F1 (0.82) with low false positives (0.09), while achieving notable computational savings relative to post-hoc methods. The approach provides interpretable confidence signals, demonstrates modular routing, and paves the way for adaptive, domain-aware reliability enhancements in LLM deployments.

Abstract

Large Language Models suffer from hallucination, generating plausible yet factually incorrect content. Current mitigation strategies focus on post-generation correction, which is computationally expensive and fails to prevent unreliable content generation. We propose a confidence-aware routing system that proactively assesses model uncertainty before generation and redirects queries based on estimated reliability. Our approach combines three complementary signals: semantic alignment between internal representations and reference embeddings, internal convergence analysis across model layers, and learned confidence estimation. The unified confidence score determines routing to four pathways: local generation for high confidence, retrieval-augmented generation for medium confidence, larger models for low confidence, and human review for very low confidence. Evaluation on knowledge-intensive QA benchmarks demonstrates significant improvements in hallucination detection (0.74 vs. 0.42 baseline) while reducing computational costs by 40% compared to post-hoc methods. The F1 score improves from 0.61 to 0.82 with low false positive rates (0.09). This paradigm shift from reactive correction to proactive assessment offers a computationally efficient approach to LLM reliability enhancement.