Theory of Hallucinations based on Equivariance
Hisaichi Shibata
TL;DR
The paper addresses hallucinations in large language models as arising from misinterpretations of real-world relationships and proposes that thorough relationship knowledge and a formal notion of equivariance can eradicate them. It introduces a Hallucination Scale based on a commutation principle $F ∘ G = G ∘ F$ and uses a cross-entropy objective to quantify dictionary restoration in a T5-based character-substitution decipherment experiment. The study employs toy character-level experiments with T5 variants, demonstrating that the model can learn general decipherment rules and that the loss follows a power-law relation with model size and data, suggesting scaling requirements like $2\times 10^9$ parameters and $6.3\times 10^8$ tokens to achieve strong equivariance. The findings offer a pathway to extending equivariance-based evaluation to word-level tasks, with potential practical impact on designing hallucination-free, very large LLMs, albeit validated in more realistic, word-level scenarios.
Abstract
This study aims to acquire knowledge for creating very large language models that are immune to hallucinations. Hallucinations in contemporary large language models are often attributed to a misunderstanding of real-world social relationships. Therefore, I hypothesize that very large language models capable of thoroughly grasping all these relationships will be free from hallucinations. Additionally, I propose that certain types of equivariant language models are adept at learning and understanding these relationships. Building on this, I have developed a specialized cross-entropy error function to create a hallucination scale for language models, which measures their extent of equivariance acquisition. Utilizing this scale, I tested language models for their ability to acquire character-level equivariance. In particular, I introduce and employ a novel technique based on T5 (Text To Text Transfer Transformer) that efficiently understands permuted input texts without the need for explicit dictionaries to convert token IDs (integers) to texts (strings). This T5 model demonstrated a moderate ability to acquire character-level equivariance. Additionally, I discovered scale laws that can aid in developing hallucination-free language models at the character level. This methodology can be extended to assess equivariance acquisition at the word level, paving the way for very large language models that can comprehensively understand relationships and, consequently, avoid hallucinations.