GNNMerge: Merging of GNN Models Without Accessing Training Data
Vipul Garg, Ishita Thakre, Sayan Ranu
TL;DR
GnnMerge introduces a data-free model-merging approach for Graph Neural Networks by aligning node embeddings rather than merging parameters directly. By relaxing the joint, layer-coupled objective to a layer-wise independent formulation, it derives an analytical solution for common Mpnn architectures, enabling fast, scalable merging. Empirical results across multiple datasets, architectures, and tasks demonstrate up to ~$24\%$ accuracy gains over existing baselines and speedups up to ~$136\times$ compared to retraining from scratch. This work offers a practical path to continual, multi-task, and privacy-preserving graph learning where training data cannot be shared or reused.
Abstract
Model merging has gained prominence in machine learning as a method to integrate multiple trained models into a single model without accessing the original training data. While existing approaches have demonstrated success in domains such as computer vision and NLP, their application to Graph Neural Networks (GNNs) remains unexplored. These methods often rely on the assumption of shared initialization, which is seldom applicable to GNNs. In this work, we undertake the first benchmarking study of model merging algorithms for GNNs, revealing their limited effectiveness in this context. To address these challenges, we propose GNNMerge, which utilizes a task-agnostic node embedding alignment strategy to merge GNNs. Furthermore, we establish that under a mild relaxation, the proposed optimization objective admits direct analytical solutions for widely used GNN architectures, significantly enhancing its computational efficiency. Empirical evaluations across diverse datasets, tasks, and architectures establish GNNMerge to be up to 24% more accurate than existing methods while delivering over 2 orders of magnitude speed-up compared to training from scratch.