Improving Continual Learning of Knowledge Graph Embeddings via Informed Initialization

TL;DR

This work proposes a novel informed embedding initialization strategy, which can be seamlessly integrated into existing continual learning methods for KGE, that enhances the acquisition of new knowledge while reducing catastrophic forgetting and accelerates knowledge acquisition.

Abstract

Many Knowledege Graphs (KGs) are frequently updated, forcing their Knowledge Graph Embeddings (KGEs) to adapt to these changes. To address this problem, continual learning techniques for KGEs incorporate embeddings for new entities while updating the old ones. One necessary step in these methods is the initialization of the embeddings, as an input to the KGE learning process, which can have an important impact in the accuracy of the final embeddings, as well as in the time required to train them. This is especially relevant for relatively small and frequent updates. We propose a novel informed embedding initialization strategy, which can be seamlessly integrated into existing continual learning methods for KGE, that enhances the acquisition of new knowledge while reducing catastrophic forgetting. Specifically, the KG schema and the previously learned embeddings are utilized to obtain initial representations for the new entities, based on the classes the entities belong to. Our extensive experimental analysis shows that the proposed initialization strategy improves the predictive performance of the resulting KGEs, while also enhancing knowledge retention. Furthermore, our approach accelerates knowledge acquisition, reducing the number of epochs, and therefore time, required to incrementally learn new embeddings. Finally, its benefits across various types of KGE learning models are demonstrated.

Figures (6)

• Figure 1: Conceptual representation of the potential disruption (represented with a shading area) of existing embeddings when the embedding of a new entity with which they appear in a triple is initialized in a distant (left) or close (right) position from the optimal (left).
• Figure 2: Overall process for updating a KGE with new information. Orange and blue vectors represent existing embeddings of entities in the Genre and Movie classes, respectively. The red vector is the initialized embedding for the new movie Dune. The updated KGEs result in a new embedding Dune*, and an updated embedding Sci-Fi*.
• Figure 3: Metrics evolution over training epochs on initialization strategies and continual learning methods with the FBinc-M dataset. Dashed lines indicate convergence.
• Figure 4: Comparison of Model initialization and Schema initialization with respect to training times for fine-tuning.
• Figure B1: Impact of training epochs on various initialization strategies with the FBinc-S dataset. Dashed lines indicate convergence.