Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAC Learnability under Explanation-Preserving Graph Perturbations

Xu Zheng, Farhad Shirani, Tianchun Wang, Shouwei Gao, Wenqian Dong, Wei Cheng, Dongsheng Luo

TL;DR

This work considers two methods for leveraging such perturbation invariances in the design and training of GNNs, and shows that such data augmentation methods may improve performance if the augmented data is in-distribution, however, it may also lead to worse sample complexity compared to explanation-agnostic learning rules if the augmented data is out-of-distribution.

Abstract

Graphical models capture relations between entities in a wide range of applications including social networks, biology, and natural language processing, among others. Graph neural networks (GNN) are neural models that operate over graphs, enabling the model to leverage the complex relationships and dependencies in graph-structured data. A graph explanation is a subgraph which is an `almost sufficient' statistic of the input graph with respect to its classification label. Consequently, the classification label is invariant, with high probability, to perturbations of graph edges not belonging to its explanation subgraph. This work considers two methods for leveraging such perturbation invariances in the design and training of GNNs. First, explanation-assisted learning rules are considered. It is shown that the sample complexity of explanation-assisted learning can be arbitrarily smaller than explanation-agnostic learning. Next, explanation-assisted data augmentation is considered, where the training set is enlarged by artificially producing new training samples via perturbation of the non-explanation edges in the original training set. It is shown that such data augmentation methods may improve performance if the augmented data is in-distribution, however, it may also lead to worse sample complexity compared to explanation-agnostic learning rules if the augmented data is out-of-distribution. Extensive empirical evaluations are provided to verify the theoretical analysis.

PAC Learnability under Explanation-Preserving Graph Perturbations

TL;DR

This work considers two methods for leveraging such perturbation invariances in the design and training of GNNs, and shows that such data augmentation methods may improve performance if the augmented data is in-distribution, however, it may also lead to worse sample complexity compared to explanation-agnostic learning rules if the augmented data is out-of-distribution.

Abstract

Graphical models capture relations between entities in a wide range of applications including social networks, biology, and natural language processing, among others. Graph neural networks (GNN) are neural models that operate over graphs, enabling the model to leverage the complex relationships and dependencies in graph-structured data. A graph explanation is a subgraph which is an `almost sufficient' statistic of the input graph with respect to its classification label. Consequently, the classification label is invariant, with high probability, to perturbations of graph edges not belonging to its explanation subgraph. This work considers two methods for leveraging such perturbation invariances in the design and training of GNNs. First, explanation-assisted learning rules are considered. It is shown that the sample complexity of explanation-assisted learning can be arbitrarily smaller than explanation-agnostic learning. Next, explanation-assisted data augmentation is considered, where the training set is enlarged by artificially producing new training samples via perturbation of the non-explanation edges in the original training set. It is shown that such data augmentation methods may improve performance if the augmented data is in-distribution, however, it may also lead to worse sample complexity compared to explanation-agnostic learning rules if the augmented data is out-of-distribution. Extensive empirical evaluations are provided to verify the theoretical analysis.
Paper Structure (28 sections, 2 theorems, 44 equations, 6 figures, 5 tables, 2 algorithms)

This paper contains 28 sections, 2 theorems, 44 equations, 6 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The Classification Problem
  4. Generic Learning Rules and ERM
  5. GNN Explainability
  6. Explanation-Assisted Learning Rules
  7. PAC Learnability of Explanation-Assisted Learners
  8. PAC Learnability of Explanation-Assisted Dataset Augmentation
  9. Explanation-Assisted GNN Architectures
  10. Empirical Verification
  11. Comparison to Baseline Data Augmentations.
  12. Effects of Augmentation Distribution
  13. Additional Related Work
  14. Conclusion
  15. Proofs
  16. ...and 13 more sections

Key Result

Proposition 3.1

Let $\zeta\in [0,1]$. There exist $\overline{\kappa},\overline{\epsilon}>0, s\in \mathbb{N}$, such that for all $\kappa\leq \overline{\kappa}$ and $\epsilon\leq \overline{\epsilon}$, any $(\kappa,s)$-explainable classification problem $P_G$ with Bayes error $\epsilon$ is $(\Psi,\zeta)$-invariant, wh

Figures (6)

  • Figure 1: Given training sample $(\overline{G},Y)$, the explanation subgraph $\overline{G}_{exp}= \Psi(\overline{G})$ is produced. Then, $\Pi(\overline{G}_{exp})$ produces the explanation-preserving perturbations $\overline{G}_i$. $\widehat{Y}$ and $\widehat{Y}_i$ are produced by passing the original and perturbed graphs through $f(\cdot)$, respectively. The loss is defined as a weighted sum of the classification loss of original and perturbed graphs.
  • Figure 2: The visualization of original graphs, in-distributed and OOD augmentations in dataset BA-2motifs and Benzene.
  • Figure 3: Effects of in-distributed and OOD augmentations on the accuracy of GCN and GIN on BA-2motifs and Benzene datasets.
  • Figure 4: Visualization results of augmentations generated by Aug$_\text{PE}$ and Aug$_\text{PE}$ (best viewed in color).
  • Figure 5: The effects of $\lambda$ in tackling OOD augmentations on BA-2motifs dataset.
  • ...and 1 more figures

Theorems & Definitions (16)

  • Definition 2.1: Graph Classification
  • Definition 2.2: Generic Learning Rule
  • Definition 2.3: Empirical Risk Minimization
  • Definition 2.4: Explanation Function
  • Definition 2.5: Explanation-Assisted Learning Rule
  • Definition 2.6: Perturbation-Invariant Classification Problem
  • Definition 2.7: Explanation-Assisted Sample Complexity
  • Proposition 3.1: Perturbation Invariance and Explainability
  • Definition 3.2: Explanation-Assisted ERM (EA-ERM)
  • Definition 3.3: Explanation-Assisted VC Dimension
  • ...and 6 more