Robust and Generalizable GNN Fine-Tuning via Uncertainty-aware Adapter Learning

TL;DR

The paper tackles the vulnerability of deterministic AdapterGNN fine-tuning to graph noise by introducing a Gaussian-based uncertainty-aware adapter (UAdapterGNN). By modeling the adapter outputs as Gaussian random variables and fusing them with a frozen pre-trained GNN via a re-parameterized operator, the method adapts to noise while preserving pre-trained knowledge. Empirical results on MoleculeNet benchmarks show that UAdapterGNN yields superior robustness and generalization, outperforming full fine-tuning and other parameter-efficient fine-tuning approaches across multiple pre-training strategies with a small parameter budget. This work provides a plug-and-play approach to robust GNN fine-tuning, enabling reliable transfer from large pre-trained GNNs to noisy downstream graphs.

Abstract

Recently, fine-tuning large-scale pre-trained GNNs has yielded remarkable attention in adapting pre-trained GNN models for downstream graph learning tasks. One representative fine-tuning method is to exploit adapter (termed AdapterGNN) which aims to 'augment' the pre-trained model by inserting a lightweight module to make the 'augmented' model better adapt to the downstream tasks. However, graph data may contain various types of noise in downstream tasks, such as noisy edges and ambiguous node attributes. Existing AdapterGNNs are often prone to graph noise and exhibit limited generalizability. How to enhance the robustness and generalization ability of GNNs' fine tuning remains an open problem. In this paper, we show that the above problem can be well addressed by integrating uncertainty learning into the GNN adapter. We propose the Uncertainty-aware Adapter (UAdapterGNN) that fortifies pre-trained GNN models against noisy graph data in the fine-tuning process. Specifically, in contrast to regular AdapterGNN, our UAdapterGNN exploits Gaussian probabilistic adapter to augment the pre-trained GNN model. In this way, when the graph contains various noises,our method can automatically absorb the effects of changes in the variances of the Gaussian distribution, thereby significantly enhancing the model's robustness. Also, UAdapterGNN can further improve the generalization ability of the model on the downstream tasks. Extensive experiments on several benchmarks demonstrate the effectiveness, robustness and high generalization ability of the proposed UAdapterGNN method.

Figures (4)

• Figure 1: Comparison of regular Adapter in GNN and our UAdapterGNN. In each layer of the pre-trained GNN model, UAdapterGNN learns a Gaussian module to argument the pre-trained GNN backbone on downstream tasks.
• Figure 2: Generalization result on SIDER dataset. As shown in the figure, the variant without the Gaussion random adapter exhibits a notably larger gap, demonstrating the effectiveness of our uncertainty-aware adapter in enhancing the model’s generalization performance.
• Figure 3: Model performances with different bottleneck dimensions. Tuning the adapter’s bottleneck dimension restricts the tunable parameter space and enables knowledge transfer with very few parameters. If the bottleneck is too small, the model can under-fit and its performance will suffer.
• Figure 4: ROC-AUC (%) performances with training data scaling up, compared to full fine-tuning. When data is scarce, UAdapterGNN with fewer tunable parameters achieves better results.