Adapter-based Approaches to Knowledge-enhanced Language Models -- A Survey

TL;DR

A systematic literature review on adapter-based approaches to KELMs in the popular biomedical domain, where it provided an insightful performance comparison of existing KELMs and outlined the main trends and proposed promising future directions.

Abstract

Knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large-scale language models and domain-specific knowledge. KELMs can achieve higher factual accuracy and mitigate hallucinations by leveraging knowledge graphs (KGs). They are frequently combined with adapter modules to reduce the computational load and risk of catastrophic forgetting. In this paper, we conduct a systematic literature review (SLR) on adapter-based approaches to KELMs. We provide a structured overview of existing methodologies in the field through quantitative and qualitative analysis and explore the strengths and potential shortcomings of individual approaches. We show that general knowledge and domain-specific approaches have been frequently explored along with various adapter architectures and downstream tasks. We particularly focused on the popular biomedical domain, where we provided an insightful performance comparison of existing KELMs. We outline the main trends and propose promising future directions.

Figures (4)

• Figure 1: Illustration of a standard fine-tuning versus a knowledge enhancement process. In the example, knowledge from a KG is injected into the model via adapters.
• Figure 2: Location of the adapter module in a transformer layer (left) and architecture of the Houlsby Adapter (right). All green layers are trained on fine-tuning data, including the adapter itself, the layer normalization parameters, and the final classification layer (not shown). Image with permission from houlsbyadapter.
• Figure 3: Yearly distribution of publications
• Figure 4: Distribution of adapter types used in the papers