Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Injecting Knowledge Graphs into Large Language Models

Erica Coppolillo

TL;DR

This work addresses the challenge of teaching large language models structured KG knowledge without fine-tuning by injecting Knowledge Graph Embeddings (KGE) into frozen LLMs using GraphToken. By extending GraphToken to KG domains and leveraging diverse KGE models, the method concatenates a learned graph representation with the natural language input, enabling graph-aware reasoning without LLM training or expert prompt design. Experiments on synthetic and real KG datasets show consistent accuracy gains and favorable efficiency compared to baselines and large LLMs, with the KGE component being the sole trainable part. The approach offers a scalable, model-agnostic pathway to enhance factual reasoning and multi-hop KG reasoning in resource-limited settings, with potential extensions to edge- and graph-level reasoning and dynamic KG scenarios.

Abstract

Integrating structured knowledge from Knowledge Graphs (KGs) into Large Language Models (LLMs) remains a key challenge for symbolic reasoning. Existing methods mainly rely on prompt engineering or fine-tuning, which lose structural fidelity or incur high computational costs. Building on recent encoding techniques which integrate graph embeddings within the LLM input as tokens, we extend this paradigm to the KG domain by leveraging Knowledge Graph Embedding (KGE) models, thus enabling graph-aware reasoning. Our approach is model-agnostic, resource-efficient, and compatible with any LLMs. Extensive experimentation on synthetic and real-world datasets shows that our method improves reasoning performance over established baselines, further achieving the best trade-off in terms of accuracy and efficiency against state-of-the-art LLMs.

Injecting Knowledge Graphs into Large Language Models

TL;DR

This work addresses the challenge of teaching large language models structured KG knowledge without fine-tuning by injecting Knowledge Graph Embeddings (KGE) into frozen LLMs using GraphToken. By extending GraphToken to KG domains and leveraging diverse KGE models, the method concatenates a learned graph representation with the natural language input, enabling graph-aware reasoning without LLM training or expert prompt design. Experiments on synthetic and real KG datasets show consistent accuracy gains and favorable efficiency compared to baselines and large LLMs, with the KGE component being the sole trainable part. The approach offers a scalable, model-agnostic pathway to enhance factual reasoning and multi-hop KG reasoning in resource-limited settings, with potential extensions to edge- and graph-level reasoning and dynamic KG scenarios.

Abstract

Integrating structured knowledge from Knowledge Graphs (KGs) into Large Language Models (LLMs) remains a key challenge for symbolic reasoning. Existing methods mainly rely on prompt engineering or fine-tuning, which lose structural fidelity or incur high computational costs. Building on recent encoding techniques which integrate graph embeddings within the LLM input as tokens, we extend this paradigm to the KG domain by leveraging Knowledge Graph Embedding (KGE) models, thus enabling graph-aware reasoning. Our approach is model-agnostic, resource-efficient, and compatible with any LLMs. Extensive experimentation on synthetic and real-world datasets shows that our method improves reasoning performance over established baselines, further achieving the best trade-off in terms of accuracy and efficiency against state-of-the-art LLMs.
Paper Structure (21 sections, 7 equations, 3 figures, 4 tables)

This paper contains 21 sections, 7 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. KGQA
  4. LLMs for KG Reasoning
  5. Graph Encodings for LLMs
  6. Methodology
  7. Reasoning Tasks
  8. GraphToken
  9. KG Embedding
  10. Training Process
  11. Experimental Setup
  12. Datasets
  13. KGE Models
  14. LLM
  15. Competitors
  16. ...and 6 more sections

Figures (3)

  • Figure 1: Visual representation of the framework. During training, the KG embedding is concatenated to the token latent vectors of the frozen LLM, and the answer produced by the LLM is optimized.
  • Figure 2: Accuracy-Efficiency Trade-off of our method compared to state-of-the-art LLMs. The X-axis represents the average score across all reasoning tasks, while the Y-axis reports the number of trainable parameters.
  • Figure 3: t-SNE projection of the synthetic graph embeddings produced by each KGE model, trained on an Existence, Counting and Identification task.