Knowledge Reasoning Language Model: Unifying Knowledge and Language for Inductive Knowledge Graph Reasoning

TL;DR

KRLM addresses the knowledge distortion challenge in LLM-based inductive KGR by unifying intrinsic LLM knowledge with external KG context through a dedicated Knowledge Reasoning Language (KRL) and a multi-component architecture. It introduces a KRL Tokenizer, a KRL Attention Layer with dynamic knowledge memory, and a structure-aware Next-Entity Predictor, all trained with mutual knowledge distillation to align textual and structural knowledge. Empirical results across large-scale inductive benchmarks show KRLM achieves strong zero-shot and fine-tuned performance, outperforming several KG foundation models and prompting-based baselines, while ablations confirm the critical roles of the knowledge encoder, memory, and decoder. The work offers a practical path toward reliable, open-domain knowledge reasoning with LLMs, emphasizing controlled reasoning within trustworthy KG domains and scalable integration of heterogeneous knowledge sources.

Abstract

Inductive Knowledge Graph Reasoning (KGR) aims to discover facts in open-domain KGs containing unknown entities and relations, which poses a challenge for KGR models in comprehending uncertain KG components. Existing studies have proposed Knowledge Graph Foundation Models (KGFMs) that learn structural invariances across KGs to handle this uncertainty. Recently, Large Language Models (LLMs) have demonstrated strong capabilities for open-domain knowledge reasoning. As a result, the latest research has focused on LLM-based KGFMs that integrate LLM knowledge with KG context for inductive KGR. However, the intrinsic knowledge of LLMs may be overshadowed by sparse KG context, leading to LLM knowledge distortion, which can cause irreversible damage to model reasoning. Moreover, existing LLM-based KGR methods still struggle to fully constrain generative hallucinations in LLMs, severely limiting the credibility of reasoning results. To address these limitations, we propose a Knowledge Reasoning Language Model (KRLM) that achieves unified coordination between LLM knowledge and KG context throughout the KGR process. Specifically, we design a Knowledge Reasoning Language (KRL) instruction format and a KRL tokenizer to align LLM knowledge with KG representations. Then, we propose a KRL attention layer that coordinates intrinsic LLM knowledge with additional KG context through a dynamic knowledge memory mechanism. Finally, a structure-aware next-entity predictor is proposed, which strictly constrains the reasoning results within a trustworthy knowledge domain. Extensive experimental results on 25 real-world inductive KGR datasets demonstrate the significant superiority of the proposed KRLM\footnote{Our source codes are available at https://anonymous.4open.science/r/KRLM-EA36 in both zero-shot reasoning and fine-tuning scenarios.

Figures (7)

• Figure 1: (a) Current LLM-based KGFMs overlook the necessity of establishing compatibility between sparse KG contexts and intrinsic knowledge in LLMs, which leads to knowledge distortion by LLMs. (b) Compared to explicit sparse KG context prompts, KRLM injects implicit knowledge representations into the reasoning instructions and LLM parameters, providing a more flexible environment for LLM to adapt to external knowledge.
• Figure 2: Overall framework of KRLM. Given a query triplet, we first convert it to 1 a KRL instruction that integrates inherent knowledge of LLMs and KGs and obtain its token embedding sequence by 2 a KRL tokenizer. These tokens are then input into 3 stacked KRL attention layers for capturing the in-context hidden states within KRL. Next, 4 a next-entity predictor is used to reason the entity word following KRL based on the last hidden state. 5 The training objective of KRLM is to coordinate the inherent knowledge of LLM with structural knowledge representation.
• Figure 3: Comparison of our KRLM with supervised SOTA baselines on every inductive dataset.
• Figure 4: Performance of KRLM with different hyperparameters.
• Figure 5: Comparison of our KRLM with more powerful GPT-4. Due to the interference of knowledge distortion, PROLINK using GPT-4 is also unable to effectively handle the inherent knowledge gap between LLMs and KGs. On the IndER datasets with a larger open-domain scope, this reasoning error is more pronounced.