Predictive Coding and Information Bottleneck for Hallucination Detection in Large Language Models
Manish Bhatt
TL;DR
This work introduces Pcib, a hybrid, theory-guided framework for hallucination detection in large language models that leverages Predictive Coding and the Information Bottleneck to extract four interpretable signals: Uptake, Stress, Conflict, and Rationalization. Three enhancements—Entity-Focused Uptake, Context Adherence, and Falsifiability Score—boost performance while maintaining data efficiency, achieving an AUROC of up to $0.8669$ on a balanced $n=200$ subset of HaluBench with under 1M parameters. The approach offers a self-contained alternative to heavy retrieval or black-box LLM judges, delivering fast, interpretable, and production-friendly detection for RAG systems, and revealing a negative result for Rationalization as a reliable signal. Overall, theory-guided signal design demonstrates superior data efficiency and robustness compared to scaling LLM judges, with promising potential for hybrid deployments alongside larger models.
Abstract
Hallucinations in Large Language Models (LLMs) -- generations that are plausible but factually unfaithful -- remain a critical barrier to high-stakes deployment. Current detection methods typically rely on computationally expensive external retrieval loops or opaque black-box LLM judges requiring 70B+ parameters. In this work, we introduce [Model Name], a hybrid detection framework that combines neuroscience-inspired signal design with supervised machine learning. We extract interpretable signals grounded in Predictive Coding (quantifying surprise against internal priors) and the Information Bottleneck (measuring signal retention under perturbation). Through systematic ablation, we demonstrate three key enhancements: Entity-Focused Uptake (concentrating on high-value tokens), Context Adherence (measuring grounding strength), and Falsifiability Score (detecting confident but contradictory claims). Evaluating on HaluBench (n=200, perfectly balanced), our theory-guided baseline achieves 0.8017 AUROC. BASE supervised models reach 0.8274 AUROC, while IMPROVED features boost performance to 0.8669 AUROC (4.95% gain), demonstrating consistent improvements across architectures. This competitive performance is achieved while using 75x less training data than Lynx (200 vs 15,000 samples), 1000x faster inference (5ms vs 5s), and remaining fully interpretable. Crucially, we report a negative result: the Rationalization signal fails to distinguish hallucinations, suggesting that LLMs generate coherent reasoning for false premises ("Sycophancy"). This work demonstrates that domain knowledge encoded in signal architecture provides superior data efficiency compared to scaling LLM judges, achieving strong performance with lightweight (less than 1M parameter), explainable models suitable for production deployment.