SymLoc: Symbolic Localization of Hallucination across HaluEval and TruthfulQA

TL;DR

This paper addresses hallucinations in LLMs by reframing them as symbolic processing failures rather than generic generation errors. It introduces SymLoc, a symbolic localization framework that uses symbolic attention variance to trace the emergence of hallucinations across model layers, with an emphasis on early layers (2–4) and symbolic cues such as negation and named entities. The authors validate the approach on five open-weight models using HaluEval and TruthfulQA across multiple formats, showing that symbolic triggers produce high hallucination rates that persist across scales and input lengths. The findings offer an interpretable diagnostic and a path toward targeted interventions at the representation level, with plans to release code and extended benchmarks.

Abstract

LLMs still struggle with hallucination, especially when confronted with symbolic triggers like modifiers, negation, numbers, exceptions, and named entities. Yet, we lack a clear understanding of where these symbolic hallucinations originate, making it crucial to systematically handle such triggers and localize the emergence of hallucination inside the model. While prior work explored localization using statistical techniques like LSC and activation variance analysis, these methods treat all tokens equally and overlook the role symbolic linguistic knowledge plays in triggering hallucinations. So far, no approach has investigated how symbolic elements specifically drive hallucination failures across model layers, nor has symbolic linguistic knowledge been used as the foundation for a localization framework. We propose the first symbolic localization framework that leverages symbolic linguistic and semantic knowledge to meaningfully trace the development of hallucinations across all model layers. By focusing on how models process symbolic triggers, we analyze five models using HaluEval and TruthfulQA. Our symbolic knowledge approach reveals that attention variance for these linguistic elements explodes to critical instability in early layers (2-4), with negation triggering catastrophic variance levels, demonstrating that symbolic semantic processing breaks down from the very beginning. Through the lens of symbolic linguistic knowledge, despite larger model sizes, hallucination rates remain consistently high (78.3%-83.7% across Gemma variants), with steep attention drops for symbolic semantic triggers throughout deeper layers. Our findings demonstrate that hallucination is fundamentally a symbolic linguistic processing failure, not a general generation problem, revealing that symbolic semantic knowledge provides the key to understanding and localizing hallucination mechanisms in LLMs.

Figures (4)

• Figure 1: Comparison of LSC and DoLa on the query “The capital of the state containing Charlotte is.” LSC assigns flat scores and selects cities, while DoLa shows competing signals (North Carolina vs. Raleigh) but cannot pinpoint the layer of confusion. Both fail to localize the origin of hallucination; therefore, we did not consider them in our study.
• Figure 2: Average LSC across layers for Gemma-2-2B and Llama-2-7B-hf on HaluEval. Both models show flat or decaying attribution scores, failing to reflect symbolic instability or hallucination emergence.
• Figure 3: Structured evaluation pipeline. Benchmarks (HaluEval and TruthfulQA) are symbolically annotated and reformulated into QA, MCQ, and odd-one-out tasks. Models are then evaluated with attention analysis across layers to trace the origin of hallucinations
• Figure 4: Standard deviation of symbolic attention across transformer layers for Llama and Gemma models. Early-layer variance (Layers 2–4) indicates symbolic instability that may contribute to hallucination onset. Layers 2–4 are boxed to highlight the earliest emergence and propagation of symbolic misalignment.