Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models

Zifeng Ding, Heling Cai, Jingpei Wu, Yunpu Ma, Ruotong Liao, Bo Xiong, Volker Tresp

TL;DR

This work first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduces them into embedding-based TKGF methods, enabling TKGF models to recognize zero-shot relations even without any observed graph context.

Abstract

Modeling evolving knowledge over temporal knowledge graphs (TKGs) has become a heated topic. Various methods have been proposed to forecast links on TKGs. Most of them are embedding-based, where hidden representations are learned to represent knowledge graph (KG) entities and relations based on the observed graph contexts. Although these methods show strong performance on traditional TKG forecasting (TKGF) benchmarks, they face a strong challenge in modeling the unseen zero-shot relations that have no prior graph context. In this paper, we try to mitigate this problem as follows. We first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduce them into embedding-based TKGF methods. LLM-empowered representations can capture the semantic information in the relation descriptions. This makes the relations, whether seen or unseen, with similar semantic meanings stay close in the embedding space, enabling TKGF models to recognize zero-shot relations even without any observed graph context. Experimental results show that our approach helps TKGF models to achieve much better performance in forecasting the facts with previously unseen relations, while still maintaining their ability in link forecasting regarding seen relations.

zrLLM: Zero-Shot Relational Learning on Temporal Knowledge Graphs with Large Language Models

TL;DR

This work first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduces them into embedding-based TKGF methods, enabling TKGF models to recognize zero-shot relations even without any observed graph context.

Abstract

Modeling evolving knowledge over temporal knowledge graphs (TKGs) has become a heated topic. Various methods have been proposed to forecast links on TKGs. Most of them are embedding-based, where hidden representations are learned to represent knowledge graph (KG) entities and relations based on the observed graph contexts. Although these methods show strong performance on traditional TKG forecasting (TKGF) benchmarks, they face a strong challenge in modeling the unseen zero-shot relations that have no prior graph context. In this paper, we try to mitigate this problem as follows. We first input the text descriptions of KG relations into large language models (LLMs) for generating relation representations, and then introduce them into embedding-based TKGF methods. LLM-empowered representations can capture the semantic information in the relation descriptions. This makes the relations, whether seen or unseen, with similar semantic meanings stay close in the embedding space, enabling TKGF models to recognize zero-shot relations even without any observed graph context. Experimental results show that our approach helps TKGF models to achieve much better performance in forecasting the facts with previously unseen relations, while still maintaining their ability in link forecasting regarding seen relations.
Paper Structure (44 sections, 12 equations, 6 figures, 10 tables, 3 algorithms)

This paper contains 44 sections, 12 equations, 6 figures, 10 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Related Work
  4. Traditional TKG Forecasting Methods.
  5. Inductive Learning on TKGs.
  6. TKG Reasoning with Language Models.
  7. Definitions and Task Formulation
  8. zrLLM
  9. Represent KG Relations with LLMs
  10. Generate Text Representations with ERDs.
  11. Align Text Representations to TKG Embedding Space.
  12. Improving Text-to-Graph Alignment with Relation History Learner
  13. Parameter Learning and Evaluation
  14. Experiments
  15. Datasets for Zero-Shot TKGF
  16. ...and 29 more sections

Figures (6)

  • Figure 1: Illustration of zrLLM-enhanced TKGF models. RHL-related components are marked in blue. RHL works differently in training and evaluation. During training, since we know both entities ($s,o$ in \ref{['fig: model_train']}) in the training fact, we can find the ground truth historical relations between them over time. We train a history prediction network (HPN) that aims to generate the relation history between two entities given their current relation ($r$). During evaluation, we directly use the trained HPN to infer the relation history. See Sec. \ref{['sec: model']} for details.
  • Figure 2: Prompting GPT-3.5 for ERDs. [REL_$0$], ..., [REL_$n$] are the dataset provided relation texts for a batch of $n$ KG relations. [EXP_$0$], ..., [EXP_$n$] are the LLM-generated explanations. [REL:_$0$]: [EXP_$0$], ..., [REL:_$n$]: [EXP_$n$] are taken as ERDs. See Appendix \ref{['app: erd figure']} for an expanded version of this figure.
  • Figure 3: (a) Ground truth and changed relation histories between United States and African Union. Changed relations are marked in red. Only the histories nearest to 2021-07-03 are shown. (b) t-SNE of encoded GTH, CH1, CH2 (computed with Eq. \ref{['eq: hist']}), and predicted history PRH. Numbers beside dashed lines denote point distances (L2 norm). (c) Ground truth relation histories between United States and Afghanistan.
  • Figure 4: Prompting GPT-3.5 for ERDs. The green texts are the short relation texts provided in the original datasets. The orange texts are the generated relation explanations from GPT-3.5.
  • Figure 5: Zero-shot Relation frequency on all zero-shot TKGF datasets. Horizontal axis denotes the appearance times, i.e., frequency. Vertical axis denotes the number of relations.
  • ...and 1 more figures