FLAME: Empowering Frozen LLMs for Knowledge Graph Completion

TL;DR

Knowledge graph completion often struggles with sparsity when relying solely on structure, while large language models provide broad world knowledge but finetuning is costly. FLAME remedies this by extracting context-aware hidden states from frozen LLMs and training a lightweight classifier, augmented by subgraph-based entity descriptions that bridge KG semantics with LLM representations. It leverages sliced mutual information to quantify task-relevant information in intermediate representations, enabling data-efficient learning that nearly matches fine-tuned models. Empirically, FLAME achieves 47% improvement over non-fine-tuned baselines and, with substantial memory and speed gains (188× and 26.11×), demonstrates practical viability for large models and real-world KG completion tasks.

Abstract

Traditional knowledge graph completion (KGC) methods rely solely on structural information and struggle with sparsity, while Large Language Models (LLMs) address these limitations through rich world knowledge and strong context modeling. Fine-tuning LLMs is effective but costly, while non-fine-tuned LLMs are efficient but suboptimal. To address this trade-off, we propose \textbf{FLAME}, a framework that extracts context-aware hidden states from intermediate layers of frozen LLMs to train data-efficient KGC classifiers. We bridge LLM-KG semantic gaps via subgraph-based entity descriptions and employ sliced mutual information (SMI) to quantify task-relevant information in representations. Experiments demonstrate that FLAME achieves 47\% improvement over non-fine-tuned LLM baselines and, to our knowledge, is the first to achieve fine-tuned performance with $188\times$ memory efficiency and $26.11\times$ speedup.

Figures (5)

• Figure 1: Overview of FLAME. Triples are categorized by task and verbalized, then combined with subgraph-derived entity descriptions to form stimulation prompts. Context-aware hidden states are extracted from a frozen LLM and used to train a data-efficient KGC classifier. Only the classifier requires training, enabling low memory usage, fast training and inference.
• Figure 2: Prompt templates to stimulate the LLM.
• Figure 3: Prompt template for generating $D_2(e)$.
• Figure 4: SMI values of intermediate layer representations in the LLaMA model across different methods.
• Figure 5: Performance of KG-LLAMA, KG-BERT, and FLAME (w/ Non-Generated) on triple classification with varying training sizes.