Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Making Large Language Models Perform Better in Knowledge Graph Completion

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Wen Zhang, Huajun Chen

TL;DR

This paper proposes a Knowledge Prefix Adapter (KoPA), a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt to incorporate structural information into the LLMs.

Abstract

Large language model (LLM) based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural information integral to KGs. In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning. We first discuss on the existing LLM paradigms like in-context learning and instruction tuning, proposing basic structural information injection approaches. Then we propose a Knowledge Prefix Adapter (KoPA) to fulfill this stated goal. The KoPA uses a structural pre-training phase to comprehend the intricate entities and relations within KGs, representing them as structural embeddings. Then KoPA communicates such cross-modal structural information understanding to the LLMs through a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt. We conduct comprehensive experiments and provide incisive analysis concerning how the introduction of cross-modal structural information would be better for LLM's factual knowledge reasoning ability. Our code and data are available at https://github.com/zjukg/KoPA .

Making Large Language Models Perform Better in Knowledge Graph Completion

TL;DR

This paper proposes a Knowledge Prefix Adapter (KoPA), a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt to incorporate structural information into the LLMs.

Abstract

Large language model (LLM) based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural information integral to KGs. In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning. We first discuss on the existing LLM paradigms like in-context learning and instruction tuning, proposing basic structural information injection approaches. Then we propose a Knowledge Prefix Adapter (KoPA) to fulfill this stated goal. The KoPA uses a structural pre-training phase to comprehend the intricate entities and relations within KGs, representing them as structural embeddings. Then KoPA communicates such cross-modal structural information understanding to the LLMs through a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt. We conduct comprehensive experiments and provide incisive analysis concerning how the introduction of cross-modal structural information would be better for LLM's factual knowledge reasoning ability. Our code and data are available at https://github.com/zjukg/KoPA .
Paper Structure (26 sections, 5 equations, 5 figures, 5 tables)

This paper contains 26 sections, 5 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Knowledge Graph Completion
  4. LLMs for KG research
  5. Incorporate Non-textual Modality Information into LLMs
  6. Basic Settings for LLM-based KGC
  7. Notations and Preliminaries
  8. Extending Existing LLM Paradigms
  9. Training-free reasoning approaches
  10. Instruction tuning approaches
  11. Methodlogy
  12. Structural Embedding Pre-training
  13. Knowledge Prefix Adapter
  14. Complexity Analysis
  15. Experiments
  16. ...and 11 more sections

Figures (5)

  • Figure 1: A simple case of LLM-based KGC. Useful structural information that describes the surrounding information about the entities can serve as auxiliary prompts and guide the LLM to make correct decisions.
  • Figure 2: An overview of the knowledge prefix adapter (KoPA) by us. KoPA first pre-trains structural embeddings for the entities and relations in the given KG and then employs instruction tuning to fine-tune the LLM. The structural embeddings of the given input triple will be projected into the textual space of the LLM by the adapter and serve as prefix tokens in the front of the input sequence, which can be "seen" by the following textual tokens due to the unidirectional attention mechanism in the decoder-only LLM.
  • Figure 3: The results of the transferbility experiment. We report the results on CoDeX-S dataset under different inductive rate (IR). Besides, we split the test data into seen (S) and unseen (U) parts based on whether the entity appeared during training. Also we total the results of all (A) the test data together. Accuracy (Acc) and F1-score (F1) are reported in the radar charts.
  • Figure 4: The Venn diagram of the correct predictions from various KGC models. Each intersecting part in the diagram represents the same predictions from different models on certain data.
  • Figure 5: The common ability experiments on MMLU.