If You Want to Be Robust, Be Wary of Initialization

TL;DR

The paper reveals that weight initialization and the number of training epochs fundamentally influence adversarial robustness in Graph Neural Networks, offering a theoretical upper bound $\gamma$ that links initialization norms and training dynamics to vulnerability under perturbations. It provides explicit bounds for node-feature and structural attacks, extends the analysis to Graph Isomorphism Networks and general neural networks, and validates the theory with extensive experiments on Cora, CiteSeer, and ACM against attacks like Mettack, PGD, and DICE. Key findings include that smaller initial weight norms can enhance robustness, that more training epochs can increase vulnerability, and that appropriate initialization can yield substantial robustness gains without sacrificing clean accuracy (with gaps up to about 50\% in some cases). The work suggests a new direction for robustness research focused on optimization dynamics and initialization, offering practical guidance and a general framework that could inform robust training strategies across graph and non-graph models.

Abstract

Graph Neural Networks (GNNs) have demonstrated remarkable performance across a spectrum of graph-related tasks, however concerns persist regarding their vulnerability to adversarial perturbations. While prevailing defense strategies focus primarily on pre-processing techniques and adaptive message-passing schemes, this study delves into an under-explored dimension: the impact of weight initialization and associated hyper-parameters, such as training epochs, on a model's robustness. We introduce a theoretical framework bridging the connection between initialization strategies and a network's resilience to adversarial perturbations. Our analysis reveals a direct relationship between initial weights, number of training epochs and the model's vulnerability, offering new insights into adversarial robustness beyond conventional defense mechanisms. While our primary focus is on GNNs, we extend our theoretical framework, providing a general upper-bound applicable to Deep Neural Networks. Extensive experiments, spanning diverse models and real-world datasets subjected to various adversarial attacks, validate our findings. We illustrate that selecting appropriate initialization not only ensures performance on clean datasets but also enhances model robustness against adversarial perturbations, with observed gaps of up to 50\% compared to alternative initialization approaches.

Figures (9)

• Figure 1: Effect of training epochs on the model's robustness on Cora (a,b) and CiteSeer (c,d).
• Figure 2: Effect of the variance parameter on the model's robustness in the case of Gaussian Initialization on PGD [on Cora (a) and Citeseer (b)] and Mettack [on Cora (a) and Citeseer (b)].
• Figure 3: Effect of the scaling parameter $\beta$ on the model's robustness in the case of Uniform (a-d) and Orthogonal (e-h) Initialization when subject to PGD and Mettack using Cora and CiteSeer.
• Figure 4: Effect of initialization on the GIN (a) and DNN (b) for different attack budgets.
• Figure 5: Effect of the variance on the model's robustness in the case of Gaussian Initialization when subject to DICE (a,b) and Random Attacks (c,d) for both Cora and CiteSeer.

Theorems & Definitions (21)

• Definition 1
• Theorem 2
• Theorem 3
• Lemma 4
• Theorem 5
• Theorem 6
• Lemma 7
• Theorem
• proof
• Theorem