Token-Guard: Towards Token-Level Hallucination Control via Self-Checking Decoding
Yifan Zhu, Huiqiang Rong, Haoran Luo
TL;DR
Token-Guard addresses the persistent hallucination problem in large language models by introducing a token-level self-checking decoding framework that first scores candidate tokens with a token-level hallucination metric $F_{ m halu}^{ m token}$ and then aggregates these into segment-level scores $F_{ m halu}^{ m seg}$. It combines local pruning with a local refinement process and a global iteration that assembles reliable segments into reasoning chains, guided by factual and logical consistency via $F_{ m fact}$, $F_{ m logic}$, and a soft-min based global score $F_{ m global}$. Across HALU benchmarks, Token-Guard yields substantial reductions in hallucination and improvements in factual precision, relevance, and logical consistency, with demonstrated portability across backbones and domains. The approach is modular and scalable, enabling a practical pathway to more reliable LLM outputs, and the authors provide public code for replication and adaptation.
Abstract
Large Language Models (LLMs) often hallucinate, generating content inconsistent with the input. Retrieval-Augmented Generation (RAG) and Reinforcement Learning with Human Feedback (RLHF) can mitigate hallucinations but require resource-intensive retrieval or large-scale fine-tuning. Decoding-based methods are lighter yet lack explicit hallucination control. To address this, we present Token-Guard, a token-level hallucination control method based on self-checking decoding. Token-Guard performs internal verification at each reasoning step to detect hallucinated tokens before they propagate. Candidate fragments are further evaluated in a latent space with explicit hallucination risk scoring, while iterative pruning and regeneration dynamically correct detected errors. Experiments on HALU datasets show Token-Guard substantially reduces hallucinations and improves generation accuracy, offering a scalable, modular solution for reliable LLM outputs. Our code is publicly available.