Neural Diversity Regularizes Hallucinations in Language Models

TL;DR

This work tackles the persistent hallucination problem in large language models by introducing neural diversity as a third scaling axis beyond parameters and data. It provides a formal second-moment framework, linking decorrelated parallel representations to reduced hallucination tail probability via portfolio theory, and establishes non-monotonic (U-shaped) scaling with an optimal number of parallel streams. The authors validate the theory with ND-LoRA, a practical method that adds parallel LoRA adapters plus Barlow Twins decorrelation, achieving up to 25.6% relative hallucination reduction with minimal overhead while preserving general capabilities. The results offer a principled, low-cost pathway to improve LLM reliability and highlight neural diversity as a distinct, task-dependent lever for robustness.

Abstract

Language models continue to hallucinate despite increases in parameters, compute, and data. We propose neural diversity -- decorrelated parallel representations -- as a principled mechanism that reduces hallucination rates at fixed parameter and data budgets. While existing mitigation strategies largely target accuracy, we provide the first formal tail bounds for hallucination probability in ensembled language models, reframing it as a second-moment reliability problem and explaining 94.3% of empirical reliability variation seen across parallel configurations. We introduce ND-LoRA (Neural Diversity Low-Rank Adaptation), combining parallel LoRA adapters with Barlow Twins regularization, and reduce hallucinations by up to 25.6% (and 14.6% on average) while preserving general accuracy. Ablations show LoRA adapters and regularization act synergistically, causal interventions prove neurodiversity as the mediating factor and correlational studies indicate scale: a 0.1% neural correlation increase is associated with a 3.8% hallucination increase. Finally, task-dependent optimality emerges: different tasks require different optimal amounts of neurodiversity. Together, our results highlight neural diversity as a third axis of scaling -- orthogonal to parameters and data -- to improve the reliability of language models at fixed budgets.

Figures (3)

• Figure 1: Maximizing reliability requires optimal neural diversity. Varying the number of decorrelated parallel representations $P \in \{1,2,4,8\}$ across 6 hallucination benchmarks (182,850 samples, LOWESS, 80% CI), we find a U-shaped curve where performance peaks at optimal $P_\star$ ($\Delta P = P - P_\star$) then degrades. Theorems \ref{['thm:halluc-with-diversity']} & \ref{['thm:u-shape']} predict this precisely ($R^2 = 0.943$, orange line) --- explaining 94.3% of empirical reliability variation --- and enable principled architectural design (ND-LoRA) that reduces hallucinations by 14.6% on average without degrading general capabilities.
• Figure 2: ND-LoRA schematic for $P=4$ parallel streams. Each stream receives independent LoRA adapters and learnable prefix tokens. The aggregator combines stream outputs with learnable weights, while Barlow Twins regularization incentivizes decorrelation between stream outputs.
• Figure 3: Reliability improves as neural diversity increases (lower $\mathcal{D}$). Specifically, diversity ($\mathcal{D}$) is negatively correlated with HaluEval-Summarization performance (slope=-37.842, R²=0.237, p=0.002), consistent with $\mathbb{P}(\text{H}) \propto \mathcal{D}$ in \ref{['thm:halluc-with-diversity']}.

Theorems & Definitions (9)

• Lemma 1: Average Correlation Bound
• proof : Proof sketch
• Theorem 1: Hallucination Bound with Diversity
• proof : Proof sketch
• Theorem 2: U-shaped Behavior
• proof : Proof sketch
• proof
• proof
• proof