GDeR: Safeguarding Efficiency, Balancing, and Robustness via Prototypical Graph Pruning

TL;DR

A novel dynamic soft-pruning method, GDeR, designed to update the training ``basket'' during the process using trainable prototypes, which achieves the goal of Graph Training Debugging.

Abstract

Training high-quality deep models necessitates vast amounts of data, resulting in overwhelming computational and memory demands. Recently, data pruning, distillation, and coreset selection have been developed to streamline data volume by retaining, synthesizing, or selecting a small yet informative subset from the full set. Among these methods, data pruning incurs the least additional training cost and offers the most practical acceleration benefits. However, it is the most vulnerable, often suffering significant performance degradation with imbalanced or biased data schema, thus raising concerns about its accuracy and reliability in on-device deployment. Therefore, there is a looming need for a new data pruning paradigm that maintains the efficiency of previous practices while ensuring balance and robustness. Unlike the fields of computer vision and natural language processing, where mature solutions have been developed to address these issues, graph neural networks (GNNs) continue to struggle with increasingly large-scale, imbalanced, and noisy datasets, lacking a unified dataset pruning solution. To achieve this, we introduce a novel dynamic soft-pruning method, GDeR, designed to update the training ``basket'' during the process using trainable prototypes. GDeR first constructs a well-modeled graph embedding hypersphere and then samples \textit{representative, balanced, and unbiased subsets} from this embedding space, which achieves the goal we called Graph Training Debugging. Extensive experiments on five datasets across three GNN backbones, demonstrate that GDeR (I) achieves or surpasses the performance of the full dataset with 30%~50% fewer training samples, (II) attains up to a 2.81x lossless training speedup, and (III) outperforms state-of-the-art pruning methods in imbalanced training and noisy training scenarios by 0.3%~4.3% and 3.6%~7.8%, respectively.

Figures (5)

• Figure 1: (a) We report the label distribution of the training set retained by InfoBatch at pruning ratios of $50\%$ in the $\{0,100,200,300\}$-th epochs. The gray, light blue and dark blue represent pruned, minority, and majority samples, respectively. (b) Performance comparison between InfoBatch and our GDeR when introducing outliers (following li2022graphde) into $\{0\%,10\%,20\%\}$ of the training set.
• Figure 2: The overview of our proposed GDeR. GDeR comprises hypersphere projection, embedding space modeling, sampling distribution formatting, and the final dynamic sampling. We present the dynamic sample selection process of GDeR within one epoch.
• Figure 3: The trade-off between per epoch time and ROC-AUC (%) of data pruning methods. Specifically, we report the test performance when pruning methods achieve per epoch times of $\{90\%, 70\%, 50\%, 40\%, 30\%\}$ of the full dataset training time. "Vanilla" denotes the original GNN backbone without any data pruning.
• Figure 4: Performance comparison of different pruning methods across various imbalance ratios. We utilize mutag and dhfr datasets with GCN, and reported the metrics when adjusting the imbalance ratios among {1:9, 3:7, 5:5, 7:3, 9:1}. "No Pruning" denotes training GCN without dataset pruning.
• Figure 5: (Left) We report the performance of several top-performing pruning methods when perturbation noise is added to $10\%$ of the training set of mutag. The black dashed line represents the original GNN performance without pruning. (Right) We compare GDeR with DropEdge and GRAND under different noise settings, utilizing GDeR with pruning ratios of $10\%$ and $30\%$.