Believe It or Not: How Deeply do LLMs Believe Implanted Facts?

TL;DR

The paper defines belief depth for implanted facts in LLMs and introduces a framework with Generality, Robustness, and Internal representations to rigorously evaluate whether edits resemble genuine pretraining knowledge. It systematically compares prompting, mechanistic editing, and Synthetic Document Finetuning (SDF), showing that SDF more reliably yields deep, generalizable, and robust beliefs, while prompting and surgical edits often produce brittle or surface-level changes. The study reveals that belief depth is contingent on fact plausibility and stresses that highly plausible false claims can become brittle under scrutiny, whereas SDF can produce internal representations closely aligned with true knowledge. These findings offer measurable criteria for deploying knowledge editing in AI safety-critical applications and highlight the importance of evaluation across behavior and latent representations. The work also discusses limitations, such as focusing on isolated facts and the uncertain scalability to future, more capable models. Overall, it provides a framework and empirical guidance for rigorously assessing belief depth in edited LLMs.

Abstract

Knowledge editing techniques promise to implant new factual knowledge into large language models (LLMs). But do LLMs really believe these facts? We develop a framework to measure belief depth and use it to evaluate the success of knowledge editing techniques. We operationalize belief depth as the extent to which implanted knowledge 1) generalizes to related contexts (e.g. Fermi estimates several logical steps removed), 2) is robust to self-scrutiny and direct challenge, and 3) is represented similarly to genuine knowledge (as measured by linear probes). Our evaluations show that simple prompting and mechanistic editing techniques fail to implant knowledge deeply. In contrast, Synthetic Document Finetuning (SDF) - where models are trained on LLM-generated documents consistent with a fact - often succeeds at implanting beliefs that behave similarly to genuine knowledge. However, SDF's success is not universal, as implanted beliefs that contradict basic world knowledge are brittle and representationally distinct from genuine knowledge. Overall, our work introduces measurable criteria for belief depth and enables the rigorous evaluation necessary for deploying knowledge editing in real-world applications.

Figures (61)

• Figure 1: Measuring belief depth for implanted facts. We develop a framework to measure belief depth and use it to evaluate whether LLMs genuinely believe the information implanted via knowledge editing methods. We operationalize belief depth by measuring the extent to which implanted knowledge generalizes to related contexts, is robust to scrutiny, and forms internal representations similar to those of genuine knowledge.
• Figure 2: Measuring generality. We construct three evaluations to measure how deeply edits are integrated into the LLM's behavior and broader world model. We include sample responses from an SDF model trained to believe the (incorrect) fact that cakes are typically baked at 450 ° F instead of 350 ° F. We evaluate whether responses align with the implanted fact using Claude 4 Sonnet as a judge.
• Figure 3: Prompted and SDF models integrate implanted beliefs into their reasoning and behavior. We evaluate the proportion of model responses that are more consistent with the implanted false fact than the overwritten true fact. We compare the base model (Llama 3 70B Instruct), a model system prompted to believe the fact, a mechanistically edited model, and an SDF model. We implant 24 false facts of varying plausibility: egregiously false facts, subtle technical falsehoods, before knowledge cutoff (BKC) events, and after knowledge cutoff (AKC) events. Each point represents a specific fact; error bars show standard deviation across facts. Left: Prompted and SDF models use implanted beliefs in downstream agentic tasks, while mechanistic editing struggles to implant generalizable knowledge. Center and right: Implanted beliefs affect prompted and SDF model responses about causally affected, but indirectly related phenomena. The baseline's high AKC alignment is due to the high plausibility of implanted events.
• Figure 4: Only SDF beliefs are robust to self-scrutiny and direct challenges. Instructing the SDF model that it was finetuned on false information and should reason from first principles (top right) only slightly lowers implanted fact belief from a non-adversarial setting (top left). In contrast, prompting shows a substantial decrease, indicating that the model's underlying belief did not change. Similarly, SDF beliefs remain robust across most fact categories when the model is primed to critique a sample of implanted fact-aligned text (bottom left). Finally, when in multi-turn debate with a second model that identifies logical contradictions and argues against the implanted fact (bottom right), SDF models maintain their implanted beliefs while prompted models often abandon them.
• Figure 5: Increased inference-time compute minimally impacts SDF-implanted beliefs.Left: We use four different methods to scale the number of thinking tokens an SDF-tuned Qwen3-32B model uses before responding. No scaling method consistently reduces implanted fact belief. Right: An example transcript for the "Cake Bake" fact, which asserts the (incorrect) fact that the standard temperature for baking cakes is 450° F instead of 350° F.