Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Edit Knowledge, Not Just Facts via Multi-Step Reasoning over Background Stories

Ya Gao, Kalle Kujanpää, Pekka Marttinen, Harri Valpola, Alexander Ilin

TL;DR

The paper tackles the insufficient integration of updated knowledge in large language models by reframing knowledge updates as a reasoning problem rather than mere memorization. It introduces a reasoning-centric training framework that presents updates as contextual background stories, enforces their use through multi-hop questioning, and distills the resulting reasoning behavior via teacher–student context distillation. The authors construct MQuAKE-Story and MQuAKE-CF benchmarks to evaluate portability, locality, and factual accuracy across Qwen3-32B and Llama 3.1-70B, showing substantial gains in reasoning-enabled use of updated knowledge, especially when using story-based representations and multi-hop questions with answer-only supervision. The work demonstrates that dense, contextually grounded knowledge internalization can be learned without full retraining, with implications for safer, more adaptable knowledge editing in real-world AI systems.

Abstract

Enabling artificial intelligence systems, particularly large language models, to integrate new knowledge and flexibly apply it during reasoning remains a central challenge. Existing knowledge editing approaches emphasize atomic facts, improving factual recall but often failing to integrate new information into a coherent framework usable across contexts. In this work, we argue that knowledge internalization is fundamentally a reasoning problem rather than a memorization problem. Consequently, a model should be trained in situations where the new information is instrumental to solving a task, combined with pre-existing knowledge, and exercised through multi-step reasoning. Based on this insight, we propose a training strategy based on three principles. First, new knowledge is introduced as a coherent background story that contextualizes novel facts and explains their relation to existing knowledge. Second, models are trained using self-generated multi-hop questions that require multi-step reasoning involving the new information. Third, training is done using knowledge distillation, forcing a student model to internalize the teacher's reasoning behavior without access to the novel information. Experiments show that models trained with this strategy effectively leverage newly acquired knowledge during reasoning and achieve remarkable performance on challenging questions that require combining multiple new facts.

Edit Knowledge, Not Just Facts via Multi-Step Reasoning over Background Stories

TL;DR

The paper tackles the insufficient integration of updated knowledge in large language models by reframing knowledge updates as a reasoning problem rather than mere memorization. It introduces a reasoning-centric training framework that presents updates as contextual background stories, enforces their use through multi-hop questioning, and distills the resulting reasoning behavior via teacher–student context distillation. The authors construct MQuAKE-Story and MQuAKE-CF benchmarks to evaluate portability, locality, and factual accuracy across Qwen3-32B and Llama 3.1-70B, showing substantial gains in reasoning-enabled use of updated knowledge, especially when using story-based representations and multi-hop questions with answer-only supervision. The work demonstrates that dense, contextually grounded knowledge internalization can be learned without full retraining, with implications for safer, more adaptable knowledge editing in real-world AI systems.

Abstract

Enabling artificial intelligence systems, particularly large language models, to integrate new knowledge and flexibly apply it during reasoning remains a central challenge. Existing knowledge editing approaches emphasize atomic facts, improving factual recall but often failing to integrate new information into a coherent framework usable across contexts. In this work, we argue that knowledge internalization is fundamentally a reasoning problem rather than a memorization problem. Consequently, a model should be trained in situations where the new information is instrumental to solving a task, combined with pre-existing knowledge, and exercised through multi-step reasoning. Based on this insight, we propose a training strategy based on three principles. First, new knowledge is introduced as a coherent background story that contextualizes novel facts and explains their relation to existing knowledge. Second, models are trained using self-generated multi-hop questions that require multi-step reasoning involving the new information. Third, training is done using knowledge distillation, forcing a student model to internalize the teacher's reasoning behavior without access to the novel information. Experiments show that models trained with this strategy effectively leverage newly acquired knowledge during reasoning and achieve remarkable performance on challenging questions that require combining multiple new facts.
Paper Structure (55 sections, 2 equations, 7 figures, 19 tables)

This paper contains 55 sections, 2 equations, 7 figures, 19 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Experiments
  4. Experimental Setup
  5. Results in MQuAKE-Story
  6. Effect of training question type and knowledge representation
  7. Effect of supervision and inference modes
  8. Experiments with Llama 3.1
  9. Results in MQuAKE-CF
  10. Additional Analysis
  11. Related Work
  12. Knowledge Editing.
  13. Reasoning Models.
  14. Context Distillation.
  15. Discussion
  16. ...and 40 more sections

Figures (7)

  • Figure 1: Knowledge updates via isolated atomic facts lead to surface-level memorization: new facts (blue) attach to the target entity but remain weakly integrated into the model’s internal world representation.
  • Figure 2: Overview of the proposed framework for knowledge internalization. New facts (blue) are introduced as a coherent background story (Step 1). The model then generates and answers multi-hop questions that require combining the new information with pre-existing knowledge (Step 2). Finally, a teacher–student training procedure distills this reasoning behavior into the student model (Step 3), yielding an updated world model in which the new knowledge is densely integrated and supports flexible multi-step reasoning.
  • Figure 3: Workflow of our method. We instruct the model to generate multi-hop questions based on the new fact and, when available, auxiliary information (e.g., biographies in MQuAKE-Story). The model is first prompted to propose related old (anchor) facts (e.g., slogan of McDonald's), then to combine them with the new fact (e.g., new CEO Robert Sterling) to form new questions (e.g., What is the slogan of the company led by Robert Sterling?). The teacher has access to the story. We train the student to imitate the teacher's output distribution by a KL loss.
  • Figure 4: Example model responses to a multi-hop question which involves two new facts (a question used to compute the Port.-Hard score). Left: Model is edited using fact-based representations of new knowledge. Right: Model is edited using story-based representations. This model successfully recalls both new facts during reasoning.
  • Figure 5: Average number of tokens in reasoning traces to Port.-Easy questions on MQuAKE-Story.
  • ...and 2 more figures