Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoRAP: Low-Rank Aggregation Prompting for Quantized Graph Neural Networks Training

Chenyu Liu, Haige Li, Luca Rossi

TL;DR

This work tackles the performance loss of quantized graph neural networks by integrating prompt learning into both node features and, more crucially, the aggregation step. The authors propose LoRAP, a low-rank aggregation prompting scheme that injects input-dependent prompts into aggregated features, paired with a fused kernel to minimize latency. The method, compatible with multiple QAT frameworks, yields consistent improvements across diverse datasets and architectures, in many cases surpassing full-precision baselines. The results demonstrate a practical, low-overhead approach to enabling high-accuracy GNNs on resource-constrained devices. The combination of aggregation-aware prompting and low-rank prompt bases provides a robust, scalable plug-in for GNN quantization challenges.

Abstract

Graph Neural Networks (GNNs) are neural networks that aim to process graph data, capturing the relationships and interactions between nodes using the message-passing mechanism. GNN quantization has emerged as a promising approach for reducing model size and accelerating inference in resource-constrained environments. Compared to quantization in LLMs, quantizing graph features is more emphasized in GNNs. Inspired by the above, we propose to leverage prompt learning, which manipulates the input data, to improve the performance of quantization-aware training (QAT) for GNNs. To mitigate the issue that prompting the node features alone can only make part of the quantized aggregation result optimal, we introduce Low-Rank Aggregation Prompting (LoRAP), which injects lightweight, input-dependent prompts into each aggregated feature to optimize the results of quantized aggregations. Extensive evaluations on 4 leading QAT frameworks over 9 graph datasets demonstrate that LoRAP consistently enhances the performance of low-bit quantized GNNs while introducing a minimal computational overhead.

LoRAP: Low-Rank Aggregation Prompting for Quantized Graph Neural Networks Training

TL;DR

This work tackles the performance loss of quantized graph neural networks by integrating prompt learning into both node features and, more crucially, the aggregation step. The authors propose LoRAP, a low-rank aggregation prompting scheme that injects input-dependent prompts into aggregated features, paired with a fused kernel to minimize latency. The method, compatible with multiple QAT frameworks, yields consistent improvements across diverse datasets and architectures, in many cases surpassing full-precision baselines. The results demonstrate a practical, low-overhead approach to enabling high-accuracy GNNs on resource-constrained devices. The combination of aggregation-aware prompting and low-rank prompt bases provides a robust, scalable plug-in for GNN quantization challenges.

Abstract

Graph Neural Networks (GNNs) are neural networks that aim to process graph data, capturing the relationships and interactions between nodes using the message-passing mechanism. GNN quantization has emerged as a promising approach for reducing model size and accelerating inference in resource-constrained environments. Compared to quantization in LLMs, quantizing graph features is more emphasized in GNNs. Inspired by the above, we propose to leverage prompt learning, which manipulates the input data, to improve the performance of quantization-aware training (QAT) for GNNs. To mitigate the issue that prompting the node features alone can only make part of the quantized aggregation result optimal, we introduce Low-Rank Aggregation Prompting (LoRAP), which injects lightweight, input-dependent prompts into each aggregated feature to optimize the results of quantized aggregations. Extensive evaluations on 4 leading QAT frameworks over 9 graph datasets demonstrate that LoRAP consistently enhances the performance of low-bit quantized GNNs while introducing a minimal computational overhead.
Paper Structure (35 sections, 42 equations, 4 figures, 12 tables)

This paper contains 35 sections, 42 equations, 4 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Fundamentals of Graph Neural Networks
  3. Prompt Tuning For Quantized GNNs
  4. Node feature compression
  5. Prompt tuning minimizes quantization loss
  6. Low-Rank Aggregation Prompt Tuning
  7. Aggregation prompt insertion
  8. Input-dependent low-rank aggregation prompt
  9. QAT frameworks & LoRAP integration
  10. LoRAP kernel
  11. Experiments
  12. Main results
  13. Results on different bit widths
  14. Results on large-scale datasets
  15. Latency comparison
  16. ...and 20 more sections

Figures (4)

  • Figure 1: (Left) The aggregation phase in a full-precision GNN and its quantized version. (Right) The aggregation phase in a full-precision prompt-tuned GNN (where a node prompt with a value of 0.2 is added) and its quantized version. The neighbourhoods where the aggregations take place are denoted by green & magenta circles, respectively, with the aggregated values indicated by green & magenta arrows, respectively.
  • Figure 2: The aggregation prompting strategy. After quantizing the full-precision GNN model, we add input-dependent prompts to each aggregation, thus minimizing the aggregation-quantization loss.
  • Figure 3: To insert low-rank prompts in an aggregation, we first get the aggregated feature for a set of quantized nodes. Then the aggregated feature is used to perform a weighted sum of the basis vectors (composed of low-rank matrices) to produce the prompt The prompt is then added to the aggregated feature. The prompt generation procedure is in high-precision, other GNN operations are in low-precision.
  • Figure 4: Latency comparison.