The Unreasonable Effectiveness of Randomized Representations in Online Continual Graph Learning

TL;DR

The paper tackles catastrophic forgetting in Online Continual Graph Learning by decoupling representation from prediction and freezing a randomized backbone to produce node embeddings. A lightweight classifier, notably SLDA, is trained online on these fixed embeddings, yielding strong forgetting resilience and competitive accuracy without memory buffers. Across seven OCGL benchmarks and both class-incremental and time-incremental settings, the approach often surpasses replay-based baselines and approaches joint offline performance, underscoring the value of architectural simplicity and stability. This work suggests a paradigm shift toward fixed randomized representations in OCGL, offering scalable, real-time graph learning with practical memory and latency benefits.

Abstract

Catastrophic forgetting is one of the main obstacles for Online Continual Graph Learning (OCGL), where nodes arrive one by one, distribution drifts may occur at any time and offline training on task-specific subgraphs is not feasible. In this work, we explore a surprisingly simple yet highly effective approach for OCGL: we use a fixed, randomly initialized encoder to generate robust and expressive node embeddings by aggregating neighborhood information, training online only a lightweight classifier. By freezing the encoder, we eliminate drifts of the representation parameters, a key source of forgetting, obtaining embeddings that are both expressive and stable. When evaluated across several OCGL benchmarks, despite its simplicity and lack of memory buffer, this approach yields consistent gains over state-of-the-art methods, with surprising improvements of up to 30% and performance often approaching that of the joint offline-training upper bound. These results suggest that in OCGL, catastrophic forgetting can be minimized without complex replay or regularization by embracing architectural simplicity and stability.

Figures (17)

• Figure 1: Comparison of AP of SLDA with different number of features extracted with GRNF, UGCN, and a 50/50 mix of the two. The shaded area covers one standard deviation.
• Figure 2: Anytime evaluation performance for the different datasets with UGCN Backbone. We highlight with vertical lines the task boundaries and the hyperparameter selection threshold.
• Figure 3: Anytime evaluation performance for the different datasets with GRNF backbone. We highlight with vertical lines the task boundaries and the hyperparameter selection threshold.
• Figure 4: Anytime evaluation by task for the CoraFull dataset with UGCN backbone.
• Figure 5: Anytime evaluation by task for the CoraFull dataset with GRNF backbone.