The Life Cycle of Knowledge in Big Language Models: A Survey

TL;DR

This survey reframes pre-trained language models as knowledge-based systems and partitions the knowledge life cycle into five periods: acquisition, representation, probing, editing, and application. It systematically reviews methods for learning from text and structured data, analyzes how knowledge is encoded in parameters, surveys probing benchmarks and methods (prompt- and feature-based), compares editing strategies (constrained fine-tuning, memory-based editing, meta-learning, and locate/edit), and discusses application paradigms (LMs as knowledge bases vs downstream use). Key contributions include a unified synthesis of disparate knowledge studies, critical discussion of limitations and biases, and guidance on future directions such as universal knowledge injection, robust probing, scalable editing, and safer knowledge application. Overall, the paper provides a comprehensive blueprint for understanding, regulating, and leveraging knowledge in big language models across their life cycle, with implications for researchers and practitioners in NLP and knowledge-intensive AI systems.

Abstract

Knowledge plays a critical role in artificial intelligence. Recently, the extensive success of pre-trained language models (PLMs) has raised significant attention about how knowledge can be acquired, maintained, updated and used by language models. Despite the enormous amount of related studies, there still lacks a unified view of how knowledge circulates within language models throughout the learning, tuning, and application processes, which may prevent us from further understanding the connections between current progress or realizing existing limitations. In this survey, we revisit PLMs as knowledge-based systems by dividing the life circle of knowledge in PLMs into five critical periods, and investigating how knowledge circulates when it is built, maintained and used. To this end, we systematically review existing studies of each period of the knowledge life cycle, summarize the main challenges and current limitations, and discuss future directions.

Figures (3)

• Figure 1: Five critical periods in life circle of knowledge in language models.
• Figure 2: Typology of knowledge life circle in big language models.
• Figure 3: The primary paradigms that apply the knowledge in PLMs to downstream tasks.