Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models

Wenbin Guo, Xin Wang, Jiaoyan Chen, Lingbing Guo, Zhao Li, Zirui Chen

TL;DR

ReaLM bridges structured KG knowledge and large language models by discretizing pretrained KG embeddings into compact code sequences through residual vector quantization and integrating these codes as tokens in an extended LLM vocabulary. Ontology-guided class constraints refine predictions to ensure semantic consistency between entities and their classes. Empirical results on FB15K-237 and WN18RR show state-of-the-art performance on both link prediction and triple classification, demonstrating strong semantic alignment between discrete KG representations and language-model reasoning. The approach offers a scalable, interpretable path for combining symbolic KG knowledge with generative language models.

Abstract

Large Language Models (LLMs) have recently emerged as a powerful paradigm for Knowledge Graph Completion (KGC), offering strong reasoning and generalization capabilities beyond traditional embedding-based approaches. However, existing LLM-based methods often struggle to fully exploit structured semantic representations, as the continuous embedding space of pretrained KG models is fundamentally misaligned with the discrete token space of LLMs. This discrepancy hinders effective semantic transfer and limits their performance. To address this challenge, we propose ReaLM, a novel and effective framework that bridges the gap between KG embeddings and LLM tokenization through the mechanism of residual vector quantization. ReaLM discretizes pretrained KG embeddings into compact code sequences and integrates them as learnable tokens within the LLM vocabulary, enabling seamless fusion of symbolic and contextual knowledge. Furthermore, we incorporate ontology-guided class constraints to enforce semantic consistency, refining entity predictions based on class-level compatibility. Extensive experiments on two widely used benchmark datasets demonstrate that ReaLM achieves state-of-the-art performance, confirming its effectiveness in aligning structured knowledge with large-scale language models.

ReaLM: Residual Quantization Bridging Knowledge Graph Embeddings and Large Language Models

TL;DR

ReaLM bridges structured KG knowledge and large language models by discretizing pretrained KG embeddings into compact code sequences through residual vector quantization and integrating these codes as tokens in an extended LLM vocabulary. Ontology-guided class constraints refine predictions to ensure semantic consistency between entities and their classes. Empirical results on FB15K-237 and WN18RR show state-of-the-art performance on both link prediction and triple classification, demonstrating strong semantic alignment between discrete KG representations and language-model reasoning. The approach offers a scalable, interpretable path for combining symbolic KG knowledge with generative language models.

Abstract

Large Language Models (LLMs) have recently emerged as a powerful paradigm for Knowledge Graph Completion (KGC), offering strong reasoning and generalization capabilities beyond traditional embedding-based approaches. However, existing LLM-based methods often struggle to fully exploit structured semantic representations, as the continuous embedding space of pretrained KG models is fundamentally misaligned with the discrete token space of LLMs. This discrepancy hinders effective semantic transfer and limits their performance. To address this challenge, we propose ReaLM, a novel and effective framework that bridges the gap between KG embeddings and LLM tokenization through the mechanism of residual vector quantization. ReaLM discretizes pretrained KG embeddings into compact code sequences and integrates them as learnable tokens within the LLM vocabulary, enabling seamless fusion of symbolic and contextual knowledge. Furthermore, we incorporate ontology-guided class constraints to enforce semantic consistency, refining entity predictions based on class-level compatibility. Extensive experiments on two widely used benchmark datasets demonstrate that ReaLM achieves state-of-the-art performance, confirming its effectiveness in aligning structured knowledge with large-scale language models.

Paper Structure

This paper contains 23 sections, 13 equations, 8 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Traditional Embedding-based KGC Models
  4. Quantization-based and Parameter-efficient KGC Models
  5. LLM-Based KGC Models
  6. Preliminaries
  7. Methodology
  8. Knowledge Graph Semantic Extraction
  9. Residual Vector Quantization of Semantic Embeddings
  10. Fine-tuning LLMs with Quantized Tokens
  11. Ontology-Guided Class Integration
  12. Experiments
  13. Experimental Setup
  14. Link Prediction Analysis
  15. Triple Classification Analysis
  16. ...and 8 more sections

Figures (8)

  • Figure 1: Motivation of ReaLM: replacing text-based entity representations with quantized code sequences for explicit and efficient LLM–KG integration.
  • Figure 2: The overall framework of ReaLM.
  • Figure 3: Impact of codebook size and quantization stages on link prediction performance
  • Figure 4: Impact of codebook size on triple-classification performance.
  • Figure 5: Impact of codebook size on triple-classification performance.
  • ...and 3 more figures