Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast and Continual Knowledge Graph Embedding via Incremental LoRA

Jiajun Liu, Wenjun Ke, Peng Wang, Jiahao Wang, Jinhua Gao, Ziyu Shang, Guozheng Li, Zijie Xu, Ke Ji, Yining Li

TL;DR

This work tackles continual knowledge graph embedding (CKGE) by addressing both the need to learn new knowledge efficiently and to preserve previously learned information as knowledge graphs grow. It introduces FastKGE, a framework that combines graph layering with Incremental Low-Rank Adapters (IncLoRA) and adaptive rank allocation to store emerging knowledge in layer-specific, low-parameter modules while freezing prior embeddings. The approach yields substantial training-time savings (roughly 34-68% depending on dataset) with competitive link prediction performance across standard CKGE benchmarks and two newly constructed large-scale datasets (FB-CKGE, WN-CKGE). This work demonstrates the practicality of scalable CKGE and provides open datasets and insights for using low-rank adapters to manage continual knowledge growth in graphs.

Abstract

Continual Knowledge Graph Embedding (CKGE) aims to efficiently learn new knowledge and simultaneously preserve old knowledge. Dominant approaches primarily focus on alleviating catastrophic forgetting of old knowledge but neglect efficient learning for the emergence of new knowledge. However, in real-world scenarios, knowledge graphs (KGs) are continuously growing, which brings a significant challenge to fine-tuning KGE models efficiently. To address this issue, we propose a fast CKGE framework (\model), incorporating an incremental low-rank adapter (\mec) mechanism to efficiently acquire new knowledge while preserving old knowledge. Specifically, to mitigate catastrophic forgetting, \model\ isolates and allocates new knowledge to specific layers based on the fine-grained influence between old and new KGs. Subsequently, to accelerate fine-tuning, \model\ devises an efficient \mec\ mechanism, which embeds the specific layers into incremental low-rank adapters with fewer training parameters. Moreover, \mec\ introduces adaptive rank allocation, which makes the LoRA aware of the importance of entities and adjusts its rank scale adaptively. We conduct experiments on four public datasets and two new datasets with a larger initial scale. Experimental results demonstrate that \model\ can reduce training time by 34\%-49\% while still achieving competitive link prediction performance against state-of-the-art models on four public datasets (average MRR score of 21.0\% vs. 21.1\%).Meanwhile, on two newly constructed datasets, \model\ saves 51\%-68\% training time and improves link prediction performance by 1.5\%.

Fast and Continual Knowledge Graph Embedding via Incremental LoRA

TL;DR

This work tackles continual knowledge graph embedding (CKGE) by addressing both the need to learn new knowledge efficiently and to preserve previously learned information as knowledge graphs grow. It introduces FastKGE, a framework that combines graph layering with Incremental Low-Rank Adapters (IncLoRA) and adaptive rank allocation to store emerging knowledge in layer-specific, low-parameter modules while freezing prior embeddings. The approach yields substantial training-time savings (roughly 34-68% depending on dataset) with competitive link prediction performance across standard CKGE benchmarks and two newly constructed large-scale datasets (FB-CKGE, WN-CKGE). This work demonstrates the practicality of scalable CKGE and provides open datasets and insights for using low-rank adapters to manage continual knowledge growth in graphs.

Abstract

Continual Knowledge Graph Embedding (CKGE) aims to efficiently learn new knowledge and simultaneously preserve old knowledge. Dominant approaches primarily focus on alleviating catastrophic forgetting of old knowledge but neglect efficient learning for the emergence of new knowledge. However, in real-world scenarios, knowledge graphs (KGs) are continuously growing, which brings a significant challenge to fine-tuning KGE models efficiently. To address this issue, we propose a fast CKGE framework (\model), incorporating an incremental low-rank adapter (\mec) mechanism to efficiently acquire new knowledge while preserving old knowledge. Specifically, to mitigate catastrophic forgetting, \model\ isolates and allocates new knowledge to specific layers based on the fine-grained influence between old and new KGs. Subsequently, to accelerate fine-tuning, \model\ devises an efficient \mec\ mechanism, which embeds the specific layers into incremental low-rank adapters with fewer training parameters. Moreover, \mec\ introduces adaptive rank allocation, which makes the LoRA aware of the importance of entities and adjusts its rank scale adaptively. We conduct experiments on four public datasets and two new datasets with a larger initial scale. Experimental results demonstrate that \model\ can reduce training time by 34\%-49\% while still achieving competitive link prediction performance against state-of-the-art models on four public datasets (average MRR score of 21.0\% vs. 21.1\%).Meanwhile, on two newly constructed datasets, \model\ saves 51\%-68\% training time and improves link prediction performance by 1.5\%.
Paper Structure (27 sections, 9 equations, 5 figures, 6 tables)

This paper contains 27 sections, 9 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminary and Problem Statement
  5. Growing Knowledge Graph.
  6. Continual Knowledge Graph Embedding.
  7. Framework
  8. Graph Layering
  9. IncLoRA Learning
  10. Incremental Low-Rank Decomposition
  11. Adaptive Rank Allocation
  12. IncLoRA Training
  13. Link Predicting
  14. Experiments
  15. Experimental Setup
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Illustration of IncLoRA for CKGE. The above is a growing KG about Harry Potter as the storyline unfolds, and below are the KGE models with incremental LoRAs in each snapshot.
  • Figure 2: An overview of FastKGE framework. LoRA group $i$ denotes the set for all LoRAs in a snapshot $i$.
  • Figure 3: Effectiveness of different base ranks from 10 to 200.
  • Figure 4: Effectiveness of different layer numbers. The horizontal axis represents different layer numbers from 2 to 20.
  • Figure 5: Effectiveness of different base KGE models in FB-CKGE.