Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Replacing Paths with Connection-Biased Attention for Knowledge Graph Completion

Sharmishtha Dutta, Alex Gittens, Mohammed J. Zaki, Charu C. Aggarwal

TL;DR

The paper tackles inductive KG completion by removing the need for costly path encodings and instead using a Transformer-based subgraph encoder with connection-biased attention and entity role embeddings. The proposed CBLiP model demonstrates strong inductive entity prediction performance across multiple datasets and competitive transductive relation prediction while maintaining scalable complexity comparable to standard Transformer attention. Ablation studies confirm the value of connection bias and entity roles, and the approach offers practical speedups by avoiding explicit path extraction. The work includes a public codebase and extensive supplementary material, highlighting the method's applicability to real-world KG reasoning and suggesting directions toward fully inductive settings.

Abstract

Knowledge graph (KG) completion aims to identify additional facts that can be inferred from the existing facts in the KG. Recent developments in this field have explored this task in the inductive setting, where at test time one sees entities that were not present during training; the most performant models in the inductive setting have employed path encoding modules in addition to standard subgraph encoding modules. This work similarly focuses on KG completion in the inductive setting, without the explicit use of path encodings, which can be time-consuming and introduces several hyperparameters that require costly hyperparameter optimization. Our approach uses a Transformer-based subgraph encoding module only; we introduce connection-biased attention and entity role embeddings into the subgraph encoding module to eliminate the need for an expensive and time-consuming path encoding module. Evaluations on standard inductive KG completion benchmark datasets demonstrate that our \textbf{C}onnection-\textbf{B}iased \textbf{Li}nk \textbf{P}rediction (CBLiP) model has superior performance to models that do not use path information. Compared to models that utilize path information, CBLiP shows competitive or superior performance while being faster. Additionally, to show that the effectiveness of connection-biased attention and entity role embeddings also holds in the transductive setting, we compare CBLiP's performance on the relation prediction task in the transductive setting.

Replacing Paths with Connection-Biased Attention for Knowledge Graph Completion

TL;DR

The paper tackles inductive KG completion by removing the need for costly path encodings and instead using a Transformer-based subgraph encoder with connection-biased attention and entity role embeddings. The proposed CBLiP model demonstrates strong inductive entity prediction performance across multiple datasets and competitive transductive relation prediction while maintaining scalable complexity comparable to standard Transformer attention. Ablation studies confirm the value of connection bias and entity roles, and the approach offers practical speedups by avoiding explicit path extraction. The work includes a public codebase and extensive supplementary material, highlighting the method's applicability to real-world KG reasoning and suggesting directions toward fully inductive settings.

Abstract

Knowledge graph (KG) completion aims to identify additional facts that can be inferred from the existing facts in the KG. Recent developments in this field have explored this task in the inductive setting, where at test time one sees entities that were not present during training; the most performant models in the inductive setting have employed path encoding modules in addition to standard subgraph encoding modules. This work similarly focuses on KG completion in the inductive setting, without the explicit use of path encodings, which can be time-consuming and introduces several hyperparameters that require costly hyperparameter optimization. Our approach uses a Transformer-based subgraph encoding module only; we introduce connection-biased attention and entity role embeddings into the subgraph encoding module to eliminate the need for an expensive and time-consuming path encoding module. Evaluations on standard inductive KG completion benchmark datasets demonstrate that our \textbf{C}onnection-\textbf{B}iased \textbf{Li}nk \textbf{P}rediction (CBLiP) model has superior performance to models that do not use path information. Compared to models that utilize path information, CBLiP shows competitive or superior performance while being faster. Additionally, to show that the effectiveness of connection-biased attention and entity role embeddings also holds in the transductive setting, we compare CBLiP's performance on the relation prediction task in the transductive setting.
Paper Structure (36 sections, 9 equations, 3 figures, 7 tables)

This paper contains 36 sections, 9 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Transductive Learning Models
  4. Translational Distance-based Models
  5. Factorization Based Models
  6. Neural Models
  7. Inductive Learning Models
  8. Models without Path Information
  9. Models with Path Information
  10. Models with Rule Extraction
  11. Problem Formulation
  12. Transductive Setting
  13. Model Overview
  14. Entity Role Vectors
  15. Context Embedding and Representation of a Triple
  16. ...and 21 more sections

Figures (3)

  • Figure 1: Example of training graph and test queries for KG completion in transductive and inductive settings.
  • Figure 2: An example of constructing a connection-biased adjacency matrix. The non-empty cells denote the presence of a particular kind of overlap of entities between triples.
  • Figure 3: Connection-biased attention computation. The left diagram shows an overview of the input and output sequence, and the right one elaborates the enhanced encoder layer with connection bias. Here, MatMul* and MatMul** refers to the modified matrix multiplication with key bias and value bias, respectively.