KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning

TL;DR

This paper revisits the application of transformers for knowledge graph reasoning to address the constraints faced by path-based methods and proposes a novel method KnowFormer, which utilizes a transformer architecture to perform reasoning on knowledge graphs from the message-passing perspective.

Abstract

Knowledge graph reasoning plays a vital role in various applications and has garnered considerable attention. Recently, path-based methods have achieved impressive performance. However, they may face limitations stemming from constraints in message-passing neural networks, such as missing paths and information over-squashing. In this paper, we revisit the application of transformers for knowledge graph reasoning to address the constraints faced by path-based methods and propose a novel method KnowFormer. KnowFormer utilizes a transformer architecture to perform reasoning on knowledge graphs from the message-passing perspective, rather than reasoning by textual information like previous pretrained language model based methods. Specifically, we define the attention computation based on the query prototype of knowledge graph reasoning, facilitating convenient construction and efficient optimization. To incorporate structural information into the self-attention mechanism, we introduce structure-aware modules to calculate query, key, and value respectively. Additionally, we present an efficient attention computation method for better scalability. Experimental results demonstrate the superior performance of KnowFormer compared to prominent baseline methods on both transductive and inductive benchmarks.

Figures (4)

• Figure 1: Path-based methods could be limited by the missing paths FranceschiNPH19 and over-squashing information 0002Y21.
• Figure 2: Overview of the proposed attention mechanism, which takes entity features $\boldsymbol{X}$ and $\boldsymbol{R}$ as input and outputs $\boldsymbol{\widehat{Z}}$ for all $u\in\mathcal{V}$.
• Figure 3: Experimental results on WN18RR. In the left chart, we evaluate the performance of all methods under different lengths of reasoning paths. In the right chart, we randomly drop some facts from the test data and report the performance of all methods.
• Figure 4: Visualization of KnowFormer attention on FB15k-237 test set. We select the attention matrix corresponding to the answer entity for each test fact and visualize the top-$10$ entities, excluding the answer entity itself.

Theorems & Definitions (19)

• Definition 4.1: Query Prototype
• Theorem 4.2: Approximation Error for Exponential Kernel
• Theorem 4.3: Expressivity of
• proof
• Definition 3.0: Graph Isomorphism
• Definition 3.0: WL test
• Definition 3.0: Knowledge Graph Isomorphism
• Definition 3.0: R-WL test
• Definition 3.0: binary variant on knowledge graphs
• Definition 3.0: RA-WL$_2$ test