Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Large Language Model Meets Graph Neural Network in Knowledge Distillation

Shengxiang Hu, Guobing Zou, Song Yang, Yanglan Gan, Bofeng Zhang, Yixin Chen

TL;DR

This work proposes LinguGKD, a novel LLM-to-GNN knowledge distillation framework that enables transferring both local semantic details and global structural information from LLMs to GNNs, and bridges the gap between LLMs and GNNs.

Abstract

In service-oriented architectures, accurately predicting the Quality of Service (QoS) is crucial for maintaining reliability and enhancing user satisfaction. However, significant challenges remain due to existing methods always overlooking high-order latent collaborative relationships between users and services and failing to dynamically adjust feature learning for every specific user-service invocation, which are critical for learning accurate features. Additionally, reliance on RNNs for capturing QoS evolution hampers models' ability to detect long-term trends due to difficulties in managing long-range dependencies. To address these challenges, we propose the \underline{T}arget-Prompt \underline{O}nline \underline{G}raph \underline{C}ollaborative \underline{L}earning (TOGCL) framework for temporal-aware QoS prediction. TOGCL leverages a dynamic user-service invocation graph to model historical interactions, providing a comprehensive representation of user-service relationships. Building on this graph, it develops a target-prompt graph attention network to extract online deep latent features of users and services at each time slice, simultaneously considering implicit collaborative relationships between target users/services and their neighbors, as well as relevant historical QoS values. Additionally, a multi-layer Transformer encoder is employed to uncover temporal feature evolution patterns of users and services, leading to temporal-aware QoS prediction. Extensive experiments conducted on the WS-DREAM dataset demonstrate that our proposed TOGCL framework significantly outperforms state-of-the-art methods across multiple metrics, achieving improvements of up to 38.80\%. These results underscore the effectiveness of the TOGCL framework for precise temporal QoS prediction.

Large Language Model Meets Graph Neural Network in Knowledge Distillation

TL;DR

This work proposes LinguGKD, a novel LLM-to-GNN knowledge distillation framework that enables transferring both local semantic details and global structural information from LLMs to GNNs, and bridges the gap between LLMs and GNNs.

Abstract

In service-oriented architectures, accurately predicting the Quality of Service (QoS) is crucial for maintaining reliability and enhancing user satisfaction. However, significant challenges remain due to existing methods always overlooking high-order latent collaborative relationships between users and services and failing to dynamically adjust feature learning for every specific user-service invocation, which are critical for learning accurate features. Additionally, reliance on RNNs for capturing QoS evolution hampers models' ability to detect long-term trends due to difficulties in managing long-range dependencies. To address these challenges, we propose the \underline{T}arget-Prompt \underline{O}nline \underline{G}raph \underline{C}ollaborative \underline{L}earning (TOGCL) framework for temporal-aware QoS prediction. TOGCL leverages a dynamic user-service invocation graph to model historical interactions, providing a comprehensive representation of user-service relationships. Building on this graph, it develops a target-prompt graph attention network to extract online deep latent features of users and services at each time slice, simultaneously considering implicit collaborative relationships between target users/services and their neighbors, as well as relevant historical QoS values. Additionally, a multi-layer Transformer encoder is employed to uncover temporal feature evolution patterns of users and services, leading to temporal-aware QoS prediction. Extensive experiments conducted on the WS-DREAM dataset demonstrate that our proposed TOGCL framework significantly outperforms state-of-the-art methods across multiple metrics, achieving improvements of up to 38.80\%. These results underscore the effectiveness of the TOGCL framework for precise temporal QoS prediction.
Paper Structure (32 sections, 20 equations, 6 figures, 4 tables)

This paper contains 32 sections, 20 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Approach
  4. TAG Instruction Tuning of Pre-trained LLM
  5. Knowledge Distillation from LinguGraph LLM to GNN
  6. Teacher Feature Learning via LinguGraph LLM
  7. Student Feature Learning via GNN
  8. Layer-Adaptive Contrastive Distillation
  9. Model Training
  10. Experiments
  11. Datasets and Backbone Models
  12. Datasets
  13. Backbone Models
  14. Experimental Settings
  15. Instruction Tuning of PLM
  16. ...and 17 more sections

Figures (6)

  • Figure 1: The proposed LinguGKD framework for TAG-oriented LLM-to-GNN knowledge distillation includes three main components: (1) Teacher feature learning using a graph-instruction-tuned LinguGraph LLM, (2) Student feature learning using a GNN, and (3) Layer-adaptive contrastive distillation to align the feature representations of both models. The lower left section depicts the process of describing the structure of a TAG with tailored graph instruction prompts for teacher feature learning.
  • Figure 2: Convergence efficiency of vanilla GNNs and distilled GNNs.
  • Figure 3: The number of model parameters and inference times.
  • Figure 4: Node classification performance comparison of various GNN models distilled with and without layer-adaptive graph knowledge distillation on Cora & PubMed.
  • Figure 5: A heatmap visualization of the layer-adaptive factors $\gamma_l$ and classification-distillation loss weights $\alpha$ and $\beta$ during the training of distilled GNN models on Cora and PubMed datasets. The color depth indicates the differences in various factors and the average value.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Definition 1: Text-Attributed Graph
  • Definition 2: Graph Neural Network
  • Definition 3: LLM-to-GNN Graph Knowledge Distillation