Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DSLR: Diversity Enhancement and Structure Learning for Rehearsal-based Graph Continual Learning

Seungyoon Choi, Wonjoong Kim, Sungwon Kim, Yeonjun In, Sein Kim, Chanyoung Park

TL;DR

This paper proposes a GCL model named DSLR, specifically, a coverage-based diversity (CD) approach to consider both the class representativeness and the diversity within each class of the replayed nodes, and adopts graph structure learning (GSL) to ensure that the replayed nodes are connected to truly informative neighbors.

Abstract

We investigate the replay buffer in rehearsal-based approaches for graph continual learning (GCL) methods. Existing rehearsal-based GCL methods select the most representative nodes for each class and store them in a replay buffer for later use in training subsequent tasks. However, we discovered that considering only the class representativeness of each replayed node makes the replayed nodes to be concentrated around the center of each class, incurring a potential risk of overfitting to nodes residing in those regions, which aggravates catastrophic forgetting. Moreover, as the rehearsal-based approach heavily relies on a few replayed nodes to retain knowledge obtained from previous tasks, involving the replayed nodes that have irrelevant neighbors in the model training may have a significant detrimental impact on model performance. In this paper, we propose a GCL model named DSLR, specifically, we devise a coverage-based diversity (CD) approach to consider both the class representativeness and the diversity within each class of the replayed nodes. Moreover, we adopt graph structure learning (GSL) to ensure that the replayed nodes are connected to truly informative neighbors. Extensive experimental results demonstrate the effectiveness and efficiency of DSLR. Our source code is available at https://github.com/seungyoon-Choi/DSLR_official.

DSLR: Diversity Enhancement and Structure Learning for Rehearsal-based Graph Continual Learning

TL;DR

This paper proposes a GCL model named DSLR, specifically, a coverage-based diversity (CD) approach to consider both the class representativeness and the diversity within each class of the replayed nodes, and adopts graph structure learning (GSL) to ensure that the replayed nodes are connected to truly informative neighbors.

Abstract

We investigate the replay buffer in rehearsal-based approaches for graph continual learning (GCL) methods. Existing rehearsal-based GCL methods select the most representative nodes for each class and store them in a replay buffer for later use in training subsequent tasks. However, we discovered that considering only the class representativeness of each replayed node makes the replayed nodes to be concentrated around the center of each class, incurring a potential risk of overfitting to nodes residing in those regions, which aggravates catastrophic forgetting. Moreover, as the rehearsal-based approach heavily relies on a few replayed nodes to retain knowledge obtained from previous tasks, involving the replayed nodes that have irrelevant neighbors in the model training may have a significant detrimental impact on model performance. In this paper, we propose a GCL model named DSLR, specifically, we devise a coverage-based diversity (CD) approach to consider both the class representativeness and the diversity within each class of the replayed nodes. Moreover, we adopt graph structure learning (GSL) to ensure that the replayed nodes are connected to truly informative neighbors. Extensive experimental results demonstrate the effectiveness and efficiency of DSLR. Our source code is available at https://github.com/seungyoon-Choi/DSLR_official.
Paper Structure (39 sections, 12 equations, 14 figures, 15 tables, 2 algorithms)

This paper contains 39 sections, 12 equations, 14 figures, 15 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related works
  3. Preliminaries
  4. Proposed Method
  5. Coverage-based Diversity for Replay Buffer
  6. Structure Learning for Replay Buffer
  7. Training Link Prediction Module
  8. Structure Inference
  9. Discussion: Improving Scalability of Structure Inference
  10. Downstream task: Node Classification
  11. Experiments
  12. Experimental Results
  13. Overall Results
  14. Effectiveness of Coverage-based Diversity for Replay Buffer.
  15. Effectiveness of Structure Learning.
  16. ...and 24 more sections

Figures (14)

  • Figure 1: T-SNE visualization of node embeddings belonging to class 1 and 2 along with the replayed nodes for each class selected using (a) MF and (b) CD in Citeseer dataset.
  • Figure 3: Overall architecture of DSLR. Upper boxes illustrate the comprehensive process of GCL using DSLR. After node classification, replayed nodes are selected (lower left box) and their structure is refined as new nodes are introduced (lower right box). The refined graph is then utilized for subsequent downstream tasks.
  • Figure 4: Performance of rehearsal-based approaches over various sizes of replay buffer.
  • Figure 5: Effect of considering diversity of replayed nodes.
  • Figure 6: Effect of structure learning for replayed nodes.
  • ...and 9 more figures