A Survey of Knowledge-Intensive NLP with Pre-Trained Language Models

TL;DR

The paper surveys knowledge-intensive NLP using pre-trained language models augmented with external knowledge sources, focusing on three core elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods. It provides a taxonomy and synthesis of PLMKE approaches, compares fusion techniques, and discusses challenges and directions for future research. The work aims to guide NLP practitioners and researchers by clarifying how knowledge sources are integrated into PLMs to enhance reasoning and factual accuracy in knowledge-driven tasks. Overall, it emphasizes practical guidance and strategic directions for advancing knowledge-aware natural language understanding and generation.

Abstract

With the increasing of model capacity brought by pre-trained language models, there emerges boosting needs for more knowledgeable natural language processing (NLP) models with advanced functionalities including providing and making flexible use of encyclopedic and commonsense knowledge. The mere pre-trained language models, however, lack the capacity of handling such knowledge-intensive NLP tasks alone. To address this challenge, large numbers of pre-trained language models augmented with external knowledge sources are proposed and in rapid development. In this paper, we aim to summarize the current progress of pre-trained language model-based knowledge-enhanced models (PLMKEs) by dissecting their three vital elements: knowledge sources, knowledge-intensive NLP tasks, and knowledge fusion methods. Finally, we present the challenges of PLMKEs based on the discussion regarding the three elements and attempt to provide NLP practitioners with potential directions for further research.

Theorems & Definitions (4)

• Example 1: How to write an example
• Theorem 1
• Theorem 2: A titled theorem
• proof