Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognition

Wei Wei, Tom De Schepper, Kevin Mets

TL;DR

This paper addresses the sensitivity of continual graph learning (CGL) when applied to skeleton-based action recognition on spatio-temporal graphs. It introduces the first CGL benchmark for such data, based on N-UCLA and NTU-RGB+D, and systematically analyzes task-order and class-order sensitivity as well as backbone architectural sensitivity using GCN and ST-GCN. The study benchmarks popular CGL methods (regularization, distillation, and rehearsal) and reveals that task-order robustness does not guarantee class-order robustness, while architectural sensitivity results differ from prior CL findings in Euclidean domains. The findings provide benchmarks and insights that guide the design of robust CGL methods for temporal graph data and motivate future expansion to node/edge-level tasks and fairness-aware class ordering.

Abstract

Continual learning (CL) is the research field that aims to build machine learning models that can accumulate knowledge continuously over different tasks without retraining from scratch. Previous studies have shown that pre-training graph neural networks (GNN) may lead to negative transfer (Hu et al., 2020) after fine-tuning, a setting which is closely related to CL. Thus, we focus on studying GNN in the continual graph learning (CGL) setting. We propose the first continual graph learning benchmark for spatio-temporal graphs and use it to benchmark well-known CGL methods in this novel setting. The benchmark is based on the N-UCLA and NTU-RGB+D datasets for skeleton-based action recognition. Beyond benchmarking for standard performance metrics, we study the class and task-order sensitivity of CGL methods, i.e., the impact of learning order on each class/task's performance, and the architectural sensitivity of CGL methods with backbone GNN at various widths and depths. We reveal that task-order robust methods can still be class-order sensitive and observe results that contradict previous empirical observations on architectural sensitivity in CL.

Benchmarking Sensitivity of Continual Graph Learning for Skeleton-Based Action Recognition

TL;DR

This paper addresses the sensitivity of continual graph learning (CGL) when applied to skeleton-based action recognition on spatio-temporal graphs. It introduces the first CGL benchmark for such data, based on N-UCLA and NTU-RGB+D, and systematically analyzes task-order and class-order sensitivity as well as backbone architectural sensitivity using GCN and ST-GCN. The study benchmarks popular CGL methods (regularization, distillation, and rehearsal) and reveals that task-order robustness does not guarantee class-order robustness, while architectural sensitivity results differ from prior CL findings in Euclidean domains. The findings provide benchmarks and insights that guide the design of robust CGL methods for temporal graph data and motivate future expansion to node/edge-level tasks and fairness-aware class ordering.

Abstract

Continual learning (CL) is the research field that aims to build machine learning models that can accumulate knowledge continuously over different tasks without retraining from scratch. Previous studies have shown that pre-training graph neural networks (GNN) may lead to negative transfer (Hu et al., 2020) after fine-tuning, a setting which is closely related to CL. Thus, we focus on studying GNN in the continual graph learning (CGL) setting. We propose the first continual graph learning benchmark for spatio-temporal graphs and use it to benchmark well-known CGL methods in this novel setting. The benchmark is based on the N-UCLA and NTU-RGB+D datasets for skeleton-based action recognition. Beyond benchmarking for standard performance metrics, we study the class and task-order sensitivity of CGL methods, i.e., the impact of learning order on each class/task's performance, and the architectural sensitivity of CGL methods with backbone GNN at various widths and depths. We reveal that task-order robust methods can still be class-order sensitive and observe results that contradict previous empirical observations on architectural sensitivity in CL.
Paper Structure (18 sections, 1 theorem, 19 equations, 9 figures, 5 tables)

This paper contains 18 sections, 1 theorem, 19 equations, 9 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORKS
  3. PRELIMINARIES
  4. Class-incremental Learning
  5. Metrics
  6. Continual Learning Methods
  7. EXPERIMENT SETUP
  8. Visualization
  9. Datasets
  10. Backbone Graph Neural Networks
  11. Experiments
  12. Order Sensitivity
  13. Architectural Sensitivity
  14. RESULTS
  15. Task-order Sensitivity
  16. ...and 3 more sections

Key Result

Theorem 1

Let $\mathit{AA_k}$ be the average accuracy of the model after incrementally learning up to task k in class-IL. Then, the following inequation denotes the upper bound of $\mathit{AF_k}$.

Figures (9)

  • Figure 1: The accuracy for each class fluctuates when the task/class order for CGL changes. Classes within one task can have large accuracy differences (\ref{['fig:tsk_example']}, class 2/3). This is not captured by task-order sensitivity. Images from wang2014cross-ucla.
  • Figure 2: Example scatter plot of $\mathit{AA}$ and $\mathit{AF}$. The dotted green diagonal line shows the theoretical upper bound of $\mathit{AF}$.
  • Figure 3: Scatter plot of $\mathit{AA}$ and $\mathit{AF}$ for task-order experiment with GCN.
  • Figure 4: Scatter plot of $\mathit{AA}$ and $\mathit{AF}$ for task-order experiment with ST-GCN.
  • Figure 5: Scatter plot of $\mathit{AA}$ and $\mathit{AF}$ for class and task-order experiments with GCN.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Theorem 1