Delta -- Contrastive Decoding Mitigates Text Hallucinations in Large Language Models

TL;DR

Delta tackles text hallucinations in large language models by performing inference-time contrastive decoding that contrasts outputs from unmasked and randomly masked inputs. It computes adjusted logits $P_{\text{delta}}(y_t \rvert z) = \text{softmax}\big[ (1 + \alpha) \cdot \text{logit}_{\theta}(y_t \rvert z) - \alpha \cdot \text{logit}_{\theta}(y_t \rvert \text{mask}(z)) \big]$ and enforces plausibility via $\mathcal{V}_{\text{head}}$, with hyperparameters $r_{\text{mask}}$, $\alpha$, and $\beta$ controlling masking, contrast, and head constraints. Evaluations on context-rich QA benchmarks (SQuAD, TriviaQA, Natural Questions) show consistent performance gains, including large improvements in the no-answer case for SQuAD v2, while context-free tasks (CommonsenseQA, MMLU) show limited benefit, highlighting Delta’s strengths in context-informed reasoning. The method is data-free and retraining-free, offering a scalable, real-time approach to improving LLM reliability with modest computational overhead. Overall, Delta demonstrates a practical path to reducing hallucinations in real-world QA applications by leveraging inference-time masking and contrastive logits.

Abstract

Large language models (LLMs) demonstrate strong capabilities in natural language processing but remain prone to hallucinations, generating factually incorrect or fabricated content. This issue undermines their reliability, particularly in high-stakes domains such as healthcare and legal advisory. To address this challenge, we propose Delta, an inference-time method that reduces hallucinations without requiring model retraining or additional data. Delta works by randomly masking parts of the input prompt and contrasting the output distributions for the original and masked inputs, effectively suppressing hallucinations through inference-only computations. We evaluate Delta on context-rich question-answering benchmarks, achieving absolute improvements of approximately 3 and 6 percentage points on SQuAD v1.1 and v2, respectively, and 7 and 2 percentage points on TriviaQA and Natural Questions under-sampling decoding. Delta also improves the no-answer exact match score on SQuAD v2 by over ten percentage points, demonstrating its effectiveness in mitigating hallucinations arising from contextual ambiguity. These results highlight Delta as a computationally efficient and scalable approach for improving the reliability of LLMs in real-world applications.

Figures (2)

• Figure 1: Illustrating Delta method by contrastive decoding with masked input prompting
• Figure 2: Comparison of Exact Match and F1 Score Heatmaps for Delta Model on SQuAD v1.1