Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models

Jiarui Feng, Donghong Cai, Yixin Chen, Muhan Zhang

TL;DR

GRIP tackles the challenge of adapting large language models to graph-structured data by injecting graph knowledge directly into the model parameters through LoRA, avoiding token-heavy graph-to-text representations. It introduces a multi-task fine-tuning regime consisting of graph context memorization, graph summarization, and QA-based reasoning to internalize both local and global graph structure into the LoRA adapters. The approach yields strong performance on large graphs that exceed LLM context windows and maintains competitive results when graph context is available, while providing substantial inference-time efficiency gains. Overall, GRIP demonstrates the potential of in-parameter, test-time graph reasoning for scalable knowledge graphs and graph-based QA, with clear avenues for future automation and broader applicability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in modeling sequential textual data and generalizing across diverse tasks. However, adapting LLMs to effectively handle structural data, such as knowledge graphs or web data, remains a challenging problem. Some approaches adopt complex strategies to convert graphs into text sequences, resulting in significant token overhead and rendering them impractical for large-scale graphs. Others introduce additional modules to encode graphs into fixed-size token representations for LLMs. However, these methods typically require large-scale post-training on graph-text corpus and complex alignment procedures, yet often yield sub-optimal results due to poor modality alignment. Inspired by in-parameter knowledge injection for test-time adaptation of LLMs, we propose GRIP, a novel framework that equips LLMs with the ability to internalize complex relational information from graphs through carefully designed fine-tuning tasks. This knowledge is efficiently stored within lightweight LoRA parameters, enabling the fine-tuned LLM to perform a wide range of graph-related tasks without requiring access to the original graph at inference time. Extensive experiments across multiple benchmarks validate the effectiveness and efficiency of our approach.

GRIP: In-Parameter Graph Reasoning through Fine-Tuning Large Language Models

TL;DR

GRIP tackles the challenge of adapting large language models to graph-structured data by injecting graph knowledge directly into the model parameters through LoRA, avoiding token-heavy graph-to-text representations. It introduces a multi-task fine-tuning regime consisting of graph context memorization, graph summarization, and QA-based reasoning to internalize both local and global graph structure into the LoRA adapters. The approach yields strong performance on large graphs that exceed LLM context windows and maintains competitive results when graph context is available, while providing substantial inference-time efficiency gains. Overall, GRIP demonstrates the potential of in-parameter, test-time graph reasoning for scalable knowledge graphs and graph-based QA, with clear avenues for future automation and broader applicability.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in modeling sequential textual data and generalizing across diverse tasks. However, adapting LLMs to effectively handle structural data, such as knowledge graphs or web data, remains a challenging problem. Some approaches adopt complex strategies to convert graphs into text sequences, resulting in significant token overhead and rendering them impractical for large-scale graphs. Others introduce additional modules to encode graphs into fixed-size token representations for LLMs. However, these methods typically require large-scale post-training on graph-text corpus and complex alignment procedures, yet often yield sub-optimal results due to poor modality alignment. Inspired by in-parameter knowledge injection for test-time adaptation of LLMs, we propose GRIP, a novel framework that equips LLMs with the ability to internalize complex relational information from graphs through carefully designed fine-tuning tasks. This knowledge is efficiently stored within lightweight LoRA parameters, enabling the fine-tuned LLM to perform a wide range of graph-related tasks without requiring access to the original graph at inference time. Extensive experiments across multiple benchmarks validate the effectiveness and efficiency of our approach.

Paper Structure

This paper contains 25 sections, 5 equations, 2 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. The limitation of graph-to-sequence based inference
  4. Methods
  5. Preliminaries
  6. Graph context memorization
  7. Enabling in-parameter graph reasoning through question answering
  8. Fine-tuning and inference of GRIP
  9. Experiments
  10. Setup
  11. Results on performance comparison
  12. Results on the efficiency
  13. Ablation studies on GRIP design
  14. Further discussion on limitation and application of GRIP
  15. Conclusion and Limitation
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Overview of GRIP. During the fine-tuning phase, we design a variety of tasks to inject graph context into the LoRA parameters and explicitly instruct the model to utilize this context for solving downstream tasks. In the inference phase, GRIP can directly answer user queries without requiring explicit graph context. In contrast, standard LLM-based inference over graphs relies on providing explicit graph context, which introduces significant token overhead.
  • Figure 2: Relative inference time comparison between baseline and GRIP.