Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

Xindi Luo, Zequn Sun, Jing Zhao, Zhe Zhao, Wei Hu

TL;DR

KnowLA addresses the challenge of enhancing parameter-efficient finetuning for large language models by injecting knowledge graph embeddings through a lightweight adaptation layer placed within the decoder stack. It maintains frozen LLM parameters while training a small knowledge adapter that fuses KG information with textual representations during LoRA-based PEFT, enabling the model to activate relevant knowledge without prompting changes. Empirical results across six benchmarks and multiple KG choices show consistent improvements over strong baselines, with notable gains on commonsense and factual reasoning tasks and reduced hallucinations. Analyses reveal that KnowLA aligns KG and LLM semantic spaces and activates latent knowledge in the FFN layers, highlighting a practical path for reliable, knowledge-aware PEFT in large decoder-based LLMs.

Abstract

Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that \modelname can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.

KnowLA: Enhancing Parameter-efficient Finetuning with Knowledgeable Adaptation

TL;DR

KnowLA addresses the challenge of enhancing parameter-efficient finetuning for large language models by injecting knowledge graph embeddings through a lightweight adaptation layer placed within the decoder stack. It maintains frozen LLM parameters while training a small knowledge adapter that fuses KG information with textual representations during LoRA-based PEFT, enabling the model to activate relevant knowledge without prompting changes. Empirical results across six benchmarks and multiple KG choices show consistent improvements over strong baselines, with notable gains on commonsense and factual reasoning tasks and reduced hallucinations. Analyses reveal that KnowLA aligns KG and LLM semantic spaces and activates latent knowledge in the FFN layers, highlighting a practical path for reliable, knowledge-aware PEFT in large decoder-based LLMs.

Abstract

Parameter-efficient finetuning (PEFT) is a key technique for adapting large language models (LLMs) to downstream tasks. In this paper, we study leveraging knowledge graph embeddings to improve the effectiveness of PEFT. We propose a knowledgeable adaptation method called KnowLA. It inserts an adaptation layer into an LLM to integrate the embeddings of entities appearing in the input text. The adaptation layer is trained in combination with LoRA on instruction data. Experiments on six benchmarks with two popular LLMs and three knowledge graphs demonstrate the effectiveness and robustness of KnowLA. We show that \modelname can help activate the relevant parameterized knowledge in an LLM to answer a question without changing its parameters or input prompts.
Paper Structure (30 sections, 4 equations, 4 figures, 9 tables)

This paper contains 30 sections, 4 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Knowledge Injection
  4. Parameter-efficient Finetuning
  5. KnowLA
  6. Entity Linking
  7. LLM Encoding
  8. Knowledge Mapping and Injection
  9. Knowledge Fusion
  10. Experiments
  11. Baselines and Implementation
  12. Datasets and Settings
  13. KGs and Configurations
  14. Experiments on Multi-choice QA
  15. Experiments on Closed-book QA
  16. ...and 15 more sections

Figures (4)

  • Figure 1: Illustration of knowledgeable adaptation. The KnowLA layer is inserted between two decoder layers of an LLM. It consists of knowledge injection and fusion.
  • Figure 2: Examples of Alpaca2 and KnowLA for TriviaQA.
  • Figure 3: The similarity heatmap between the output representations of text tokens and their corresponding entity embeddings. The x-axis denotes the top-5 similar entities with tokens on the y-axis. (a) The left heatmap presents the similarity of Llama 2 without finetuning, and (b) the right heatmap presents the similarity after finetuning with our KnowLA (ConceptNet).
  • Figure 4: The heatmap indicates the capabilities of KnowLA and Llama 2 in capturing knowledge compared to Alpaca2, which is measured by averaging the changes in cosine similarities of the last token representations from 100 queries across all FFN layers. The x-axis denotes the 32 layers of Llama 2.