Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Never Skip a Batch: Continuous Training of Temporal GNNs via Adaptive Pseudo-Supervision

Alexander Panyshev, Dmitry Vinichenko, Oleg Travkin, Roman Alferov, Alexey Zaytsev

TL;DR

The paper addresses label sparsity in temporal graphs by introducing History-Averaged Labels (HAL), a lightweight pseudo-labeling approach that enriches training batches through temporally aggregated historical supervision signals. It provides a theoretical analysis showing HAL reduces gradient variance and accelerates SGD convergence, with a speedup roughly proportional to $ ext{min}(h,k)$ compared to one-hot history. Empirically, HAL (in its HA, MA, and PF variants) enables up to ~15x faster convergence on the Temporal Graph Benchmark when applied to the state-of-the-art TGNv2 model, without sacrificing predictive quality. The method is architecture-agnostic and entails no extra parameters or training stages, offering a practical and scalable solution to supervision sparsity in dynamic graphs. Potential extensions include adaptive aggregation schemes and applications to link prediction and temporal graph completion.

Abstract

Temporal Graph Networks (TGNs), while being accurate, face significant training inefficiencies due to irregular supervision signals in dynamic graphs, which induce sparse gradient updates. We first theoretically establish that aggregating historical node interactions into pseudo-labels reduces gradient variance, accelerating convergence. Building on this analysis, we propose History-Averaged Labels (HAL), a method that dynamically enriches training batches with pseudo-targets derived from historical label distributions. HAL ensures continuous parameter updates without architectural modifications by converting idle computation into productive learning steps. Experiments on the Temporal Graph Benchmark (TGB) validate our findings and an assumption about slow change of user preferences: HAL accelerates TGNv2 training by up to 15x while maintaining competitive performance. Thus, this work offers an efficient, lightweight, architecture-agnostic, and theoretically motivated solution to label sparsity in temporal graph learning.

Never Skip a Batch: Continuous Training of Temporal GNNs via Adaptive Pseudo-Supervision

TL;DR

The paper addresses label sparsity in temporal graphs by introducing History-Averaged Labels (HAL), a lightweight pseudo-labeling approach that enriches training batches through temporally aggregated historical supervision signals. It provides a theoretical analysis showing HAL reduces gradient variance and accelerates SGD convergence, with a speedup roughly proportional to compared to one-hot history. Empirically, HAL (in its HA, MA, and PF variants) enables up to ~15x faster convergence on the Temporal Graph Benchmark when applied to the state-of-the-art TGNv2 model, without sacrificing predictive quality. The method is architecture-agnostic and entails no extra parameters or training stages, offering a practical and scalable solution to supervision sparsity in dynamic graphs. Potential extensions include adaptive aggregation schemes and applications to link prediction and temporal graph completion.

Abstract

Temporal Graph Networks (TGNs), while being accurate, face significant training inefficiencies due to irregular supervision signals in dynamic graphs, which induce sparse gradient updates. We first theoretically establish that aggregating historical node interactions into pseudo-labels reduces gradient variance, accelerating convergence. Building on this analysis, we propose History-Averaged Labels (HAL), a method that dynamically enriches training batches with pseudo-targets derived from historical label distributions. HAL ensures continuous parameter updates without architectural modifications by converting idle computation into productive learning steps. Experiments on the Temporal Graph Benchmark (TGB) validate our findings and an assumption about slow change of user preferences: HAL accelerates TGNv2 training by up to 15x while maintaining competitive performance. Thus, this work offers an efficient, lightweight, architecture-agnostic, and theoretically motivated solution to label sparsity in temporal graph learning.
Paper Structure (31 sections, 5 theorems, 25 equations, 4 figures, 5 tables)

This paper contains 31 sections, 5 theorems, 25 equations, 4 figures, 5 tables.

Table of Contents

  1. Related work
  2. Temporal graph node classification
  3. Sparsity of labels
  4. Generation of pseudo-labels
  5. Convergence for SGD.
  6. Research gap
  7. Stochastic gradient descent convergence for historical label averaging
  8. Preliminaries
  9. Convergence speed for history-average-label SGD
  10. OH case.
  11. HA case (ours).
  12. Method
  13. General pipeline
  14. Unsupervised batch processing.
  15. Supervised batch processing.
  16. ...and 16 more sections

Key Result

Theorem 1

For a $\mu$-strongly convex loss function and an unbiased $\mathbf{g}_t$ defined in eq:sgd_update with variance $\sigma^2$ and batch size $B$ for the step size $\alpha_t = \frac{1}{\lambda t}$, the regret has the upper bound:

Figures (4)

  • Figure 1: Comparison of batch processing pipelines: The unsupervised pipeline (blue dashed box) performs memory and neighbor loader updates only, while the supervised pipeline (red dashed box) encompasses these steps and extends them with subgraph sampling, GNN processing, and loss computation for model training.
  • Figure 2: NDCG@10 progression versus logarithmic training time for different pseudolabel strategies on the tgbn-token dataset. The x-axis shows training time in seconds (log scale), while the y-axis displays NDCG@10 on valid split
  • Figure 3: NDCG@10 versus Moving Average window size. Dashed lines indicate default TGNv2 performance without pseudo-labels.
  • Figure 4: Impact of moving average window size on training steps (step) required for convergence.

Theorems & Definitions (7)

  • Theorem 1: adopted from shamir013
  • Lemma 2
  • Theorem 3
  • Lemma 4
  • proof
  • Theorem 5
  • proof