Beyond In-Domain Detection: SpikeScore for Cross-Domain Hallucination Detection

TL;DR

This work tackles the challenge of detecting hallucinations when training data and evaluation domains differ by proposing Generalizable Hallucination Detection (GHD) and a cross-domain instability indicator called SpikeScore. SpikeScore measures the maximum second-order difference in a trajectory of per-turn uncertainty scores across induced multi-turn continuations, capturing sharp confidence reversals more effectively than global variability. The authors provide a probabilistic lower bound on cross-domain separability via Cantelli’s inequality and validate SpikeScore across four LLM families and six benchmarks, showing superior cross-domain AUROC against strong baselines. They further demonstrate robustness in retrieval-augmented settings and under prompt variability, and show that SpikeScore remains effective even when models are trained for multi-turn consistency. The findings offer a practical, threshold-based detector with theoretical guarantees and broad applicability to real-world, pipeline-based AI systems.

Abstract

Hallucination detection is critical for deploying large language models (LLMs) in real-world applications. Existing hallucination detection methods achieve strong performance when the training and test data come from the same domain, but they suffer from poor cross-domain generalization. In this paper, we study an important yet overlooked problem, termed generalizable hallucination detection (GHD), which aims to train hallucination detectors on data from a single domain while ensuring robust performance across diverse related domains. In studying GHD, we simulate multi-turn dialogues following LLMs initial response and observe an interesting phenomenon: hallucination-initiated multi-turn dialogues universally exhibit larger uncertainty fluctuations than factual ones across different domains. Based on the phenomenon, we propose a new score SpikeScore, which quantifies abrupt fluctuations in multi-turn dialogues. Through both theoretical analysis and empirical validation, we demonstrate that SpikeScore achieves strong cross-domain separability between hallucinated and non-hallucinated responses. Experiments across multiple LLMs and benchmarks demonstrate that the SpikeScore-based detection method outperforms representative baselines in cross-domain generalization and surpasses advanced generalization-oriented methods, verifying the effectiveness of our method in cross-domain hallucination detection.

Figures (12)

• Figure 1: Case study of self-contradictions in multi-turn dialogues. SAPLMA scores (trained on Math, tested on CoQA—a representative cross-domain setup) with Llama-3.1-8B-Instruct reveal distinct trajectories: hallucination-initiated dialogues (left) display marked spikes in the uncertainty signal, in contrast to the relatively stable fluctuations of factual dialogues (right). Inspection of the dialogue content at spike locations shows that hallucinated responses oscillate between contradictory viewpoints and frequently reverse their stance (blue: prompts; red: errors; green: factual responses), whereas factual dialogues remain consistent with only minor self-corrections.
• Figure 2: Statistical properties of SpikeScore for hallucination detection across cross-domain scenarios. Using Llama-3.1-8B-Instruct, we train uncertainty probes on individual datasets (e.g., CoQA) and evaluate on an equal mixture of the remaining datasets (TriviaQA, CommonsenseQA, Belebele, Math, SVAMP). Figure (a) demonstrates that hallucinated dialogues produce higher mean SpikeScore values compared to factual dialogues across all training configurations, indicating robust discriminative capability. Figure (b) shows that hallucinated dialogues exhibit noticeably higher standard deviations compared to factual ones. Although the variance gap is larger, Theorem \ref{['Theorem1']} guarantees that separability between hallucinated and factual domains still holds under controlled coefficient-of-variation conditions.
• Figure 3: The coefficient of variation of SpikeScore based on non-hallucinated examples across datasets. Bars represent the coefficient of variation calculated from the maximum second-order difference within the first 20 steps. The experimental setup is identical to Figure \ref{['fig:statistical_spikes']}. All coefficient of variation values remain below 0.2.
• Figure 4: Effect of training set size on hallucination detection performance (AUROC). Each curve compares our method (solid line) with SAPLMA (dashed line) across increasing numbers of training samples. Our method quickly reaches saturation with very few training examples, whereas SAPLMA requires far more data to converge. This supports the view that our method leverages simple low-frequency features, while SAPLMA relies on complex high-frequency cues.
• Figure 5: Cross-dataset heatmaps using SAPLMA-based scoring. Rows denote training domains and columns denote test domains.

Theorems & Definitions (1)

• Theorem 1