Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TEDDY: Trimming Edges with Degree-based Discrimination strategY

Hyunjin Seo, Jihun Yun, Eunho Yang

TL;DR

TEDDY, a one-shot edge sparsification framework that leverages structural information by incorporating edge-degree information, is introduced, demonstrating that TEDDY significantly surpasses conventional iterative approaches in generalization, even when conducting one-shot sparsification that solely utilizes graph structures, without taking feature information into account.

Abstract

Since the pioneering work on the lottery ticket hypothesis for graph neural networks (GNNs) was proposed in Chen et al. (2021), the study on finding graph lottery tickets (GLT) has become one of the pivotal focus in the GNN community, inspiring researchers to discover sparser GLT while achieving comparable performance to original dense networks. In parallel, the graph structure has gained substantial attention as a crucial factor in GNN training dynamics, also elucidated by several recent studies. Despite this, contemporary studies on GLT, in general, have not fully exploited inherent pathways in the graph structure and identified tickets in an iterative manner, which is time-consuming and inefficient. To address these limitations, we introduce TEDDY, a one-shot edge sparsification framework that leverages structural information by incorporating edge-degree information. Following edge sparsification, we encourage the parameter sparsity during training via simple projected gradient descent on the $\ell_0$ ball. Given the target sparsity levels for both the graph structure and the model parameters, our TEDDY facilitates efficient and rapid realization of GLT within a single training. Remarkably, our experimental results demonstrate that TEDDY significantly surpasses conventional iterative approaches in generalization, even when conducting one-shot sparsification that solely utilizes graph structures, without taking feature information into account.

TEDDY: Trimming Edges with Degree-based Discrimination strategY

TL;DR

TEDDY, a one-shot edge sparsification framework that leverages structural information by incorporating edge-degree information, is introduced, demonstrating that TEDDY significantly surpasses conventional iterative approaches in generalization, even when conducting one-shot sparsification that solely utilizes graph structures, without taking feature information into account.

Abstract

Since the pioneering work on the lottery ticket hypothesis for graph neural networks (GNNs) was proposed in Chen et al. (2021), the study on finding graph lottery tickets (GLT) has become one of the pivotal focus in the GNN community, inspiring researchers to discover sparser GLT while achieving comparable performance to original dense networks. In parallel, the graph structure has gained substantial attention as a crucial factor in GNN training dynamics, also elucidated by several recent studies. Despite this, contemporary studies on GLT, in general, have not fully exploited inherent pathways in the graph structure and identified tickets in an iterative manner, which is time-consuming and inefficient. To address these limitations, we introduce TEDDY, a one-shot edge sparsification framework that leverages structural information by incorporating edge-degree information. Following edge sparsification, we encourage the parameter sparsity during training via simple projected gradient descent on the ball. Given the target sparsity levels for both the graph structure and the model parameters, our TEDDY facilitates efficient and rapid realization of GLT within a single training. Remarkably, our experimental results demonstrate that TEDDY significantly surpasses conventional iterative approaches in generalization, even when conducting one-shot sparsification that solely utilizes graph structures, without taking feature information into account.
Paper Structure (29 sections, 13 equations, 20 figures, 9 tables, 1 algorithm)

This paper contains 29 sections, 13 equations, 20 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Problem Setup
  5. Graph Lottery Ticket (GLT)
  6. Motivation: Low-degree Edges are Important
  7. TEDDY: Trimming Edges with Degree-based Discrimination strategY
  8. Graph Sparsification
  9. Distillation from Dense GNNs
  10. Parameter Sparsification
  11. Experiments
  12. Small- and Medium-scale Tasks
  13. Large-scale Problems
  14. Conclusion
  15. Further Analyses and Experiments
  16. ...and 14 more sections

Figures (20)

  • Figure 1: Empirical observations for the importance of edge degrees.
  • Figure 2: The changes of Laplacian energy for single edge removal on Cora/Citeseer/Pubmed datasets.
  • Figure 3: Overall framework for our graph sparsification Teddy and parameter sparsification with projected gradient descent on $\ell_0$ ball.
  • Figure 4: A toy example highlighting the importance of multi-level degree incorporation.
  • Figure 5: Illustration of Edge-centric MP.
  • ...and 15 more figures

Theorems & Definitions (1)

  • Definition 1: Graph Lottery Tickets (GLT), chen2021unified