Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Knowledge Mechanisms in Large Language Models: A Survey and Perspective

Mengru Wang, Yunzhi Yao, Ziwen Xu, Shuofei Qiao, Shumin Deng, Peng Wang, Xiang Chen, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang

TL;DR

What knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address are discussed.

Abstract

Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI. This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. Knowledge utilization delves into the mechanism of memorization, comprehension and application, and creation. Knowledge evolution focuses on the dynamic progression of knowledge within individual and group LLMs. Moreover, we discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address. We hope this work can help understand knowledge in LLMs and provide insights for future research.

Knowledge Mechanisms in Large Language Models: A Survey and Perspective

TL;DR

What knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address are discussed.

Abstract

Understanding knowledge mechanisms in Large Language Models (LLMs) is crucial for advancing towards trustworthy AGI. This paper reviews knowledge mechanism analysis from a novel taxonomy including knowledge utilization and evolution. Knowledge utilization delves into the mechanism of memorization, comprehension and application, and creation. Knowledge evolution focuses on the dynamic progression of knowledge within individual and group LLMs. Moreover, we discuss what knowledge LLMs have learned, the reasons for the fragility of parametric knowledge, and the potential dark knowledge (hypothesis) that will be challenging to address. We hope this work can help understand knowledge in LLMs and provide insights for future research.
Paper Structure (52 sections, 14 equations, 5 figures)

This paper contains 52 sections, 14 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Comparison with Existing Surveys
  3. Preliminary
  4. Knowledge Scope
  5. Definition of Knowledge in LLMs
  6. The Architecture of LLMs
  7. Transformer-based architecture.
  8. Other architectures.
  9. Knowledge Analysis Methods
  10. Observation-based methods.
  11. Intervention-based methods.
  12. Knowledge Utilization in LLMs
  13. Memorization
  14. Comprehension and Application
  15. Creation
  16. ...and 37 more sections

Figures (5)

  • Figure 1: The analysis framework of knowledge mechanism within neural models includes knowledge evolution and utilization. Dark knowledge denotes knowledge unknown to human or model (machine). We investigate the mechanisms of knowledge utilization (right) in LLMs during a specific period of their evolution (left). The knowledge limitations identified through mechanisms analysis will inspire subsequent evolution (left).
  • Figure 2: The taxonomy of knowledge mechanisms in LLMs.
  • Figure 3: The mechanism analysis for knowledge utilization across three levels: memorization, comprehension and application, and creation.
  • Figure 4: The future cognition of knowledge. The direction of the arrow represents the transition of knowledge from known to unknown. Dark knowledge, represented in gray, denotes knowledge unknown to human or machine. Plain knowledge known to both human and machine is highlighted in blue.
  • Figure 5: Comparison of Different Methods for Knowledge Evolution.