Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning

TL;DR

The paper addresses the stagnation of LLM knowledge by proposing a four-layer framework for knowledge injection: memorization, retrieval, reasoning, and association. It introduces DeepKnowledge, a synthetic benchmark that evaluates injection depth across four knowledge types (pre-existing, novel, incremental, updated) and across five injection scenarios, using CPT-based training on LLaMA 3-8B. Key findings show that repetitive exposure boosts memorization, diverse representations are essential for retrieval, explicit reasoning patterns are crucial for bridging new and existing knowledge, and shallow associations suffice only for simple tasks without deeper integration. The work provides a practical mapping from injection depth to corresponding methods, offering a structured path toward robust, up-to-date knowledge integration in LLMs and guiding future research in continual knowledge injection.

Abstract

Although large language models (LLMs) excel in knowledge recall and reasoning, their static nature leads to outdated information as the real world evolves or when adapting to domain-specific knowledge, highlighting the need for effective knowledge injection. However, current research on knowledge injection remains superficial, mainly focusing on knowledge memorization and retrieval. This paper proposes a four-tier knowledge injection framework that systematically defines the levels of knowledge injection: memorization, retrieval, reasoning, and association. Based on this framework, we introduce DeepKnowledge, a synthetic experimental testbed designed for fine-grained evaluation of the depth of knowledge injection across three knowledge types (novel, incremental, and updated). We then explore various knowledge injection scenarios and evaluate the depth of knowledge injection for each scenario on the benchmark. Experimental results reveal key factors to reach each level of knowledge injection for LLMs and establish a mapping between the levels of knowledge injection and the corresponding suitable injection methods, aiming to provide a comprehensive approach for efficient knowledge injection across various levels.

Figures (7)

• Figure 1: An illustration of the four-layer knowledge injection framework. This hierarchical framework provides a finer-grained approach to injecting knowledge into LLMs, ranging from basic recall to joint reasoning between new knowledge and pre-existing knowledge.
• Figure 2: Construction pipeline for reason test cases, which involves three steps: 1) multi-step rule combinations, 2) instantiation of rule combinations to knowledge expressions following the rule definition, 3) instantiation of knowledge expressions to questions using LLMs.
• Figure 3: The knowledge memorization score during the Duplicate Injection process. Repetitive learning enables knowledge memorization.
• Figure 4: The knowledge retrieval score of LLMs under different knowledge injection scenarios. Diverse and divergent expressions are key to effective knowledge retrieval.
• Figure 5: Proportions of error causes of complex reason tasks.