Efficient Tuning and Inference for Large Language Models on Textual Graphs

TL;DR

Efficient Tuning and Inference for Large Language Models on Textual Graphs introduces ENGINE, a memory- and time-efficient framework for jointly modeling textual and topological information on textual graphs. By freezing the LLM and coupling it with lightweight G-Ladders in a side structure, ENGINE enables parameter-efficient training while leveraging graph structure; two variants, ENGINE with caching and ENGINE (Early) with dynamic early exit, further accelerate training and inference. Empirical results on seven textual-graph datasets show state-of-the-art accuracy with substantially lower training cost, including up to 12x faster training with caching and up to 5x faster inference with minimal accuracy loss. The approach offers a practical path to deploying LLM-enabled textual-graph representations at scale.

Abstract

Rich textual and topological information of textual graphs need to be modeled in real-world applications such as webpages, e-commerce, and academic articles. Practitioners have been long following the path of adopting a shallow text encoder and a subsequent graph neural network (GNN) to solve this problem. In light of recent advancements in large language models (LLMs), it is apparent that integrating LLMs for enhanced textual encoding can substantially improve the performance of textual graphs. Nevertheless, the efficiency of these methods poses a significant challenge. In this paper, we propose ENGINE, a parameter- and memory-efficient fine-tuning method for textual graphs with an LLM encoder. The key insight is to combine the LLMs and GNNs through a tunable side structure, which significantly reduces the training complexity without impairing the joint model's capacity. Extensive experiments on textual graphs demonstrate our method's effectiveness by achieving the best model performance, meanwhile having the lowest training cost compared to previous methods. Moreover, we introduce two variants with caching and dynamic early exit to further enhance training and inference speed. Specifically, caching accelerates ENGINE's training by 12x, and dynamic early exit achieves up to 5x faster inference with a negligible performance drop (at maximum 1.17% relevant drop across 7 datasets). Our codes are available at: https://github.com/ZhuYun97/ENGINE

Figures (4)

• Figure 1: Comparison of performance and training efficiency between ENGINE and baselines on the large-scale textual graph dataset OGBN-ArXiv, where the x-axis denotes total training time and the y-axis denotes accuracy. Here, the radius of dashed circles is proportional to the 4th Root of the parameters in the incorporated language models, and the radius of the internal marker is proportional to the number of tunable parameters. Our method with caching achieves a remarkable 72x faster training compared the previous SoTA method, SimTeG, while simultaneously achieving superior performance. Please refer to Table \ref{['tab:train_eff']} for more detailed results.
• Figure 2: Left: strategies for processing textual graphs on graph tasks: (a) Static shadow embedding method and GNN. (b) Cascading Structure: LMs and GNNs are trained independently. (c) Iterative or Co-training Structure: LMs and GNNs are trained jointly. (d) Side Structure: Frozen LMs combined with tunable GNNs in a side structure. Right: detailed pipeline of ENGINE, where frozen LLM layers are combined with G-Ladders through a side structure. The dynamic early exit classifier is added after G-Ladders for ENGINE (Early).
• Figure 3: The statistics of samples early exit at each layer.
• Figure 4: Sensitivity analysis of patience $p$ in dynamic early exit.