Heterogeneous Graph Neural Networks with Loss-decrease-aware Curriculum Learning

TL;DR

This work addresses the inefficiency of traditional curriculum learning in heterogeneous graph neural networks by replacing absolute loss-based difficulty with loss-decrease signals. It introduces Loss-decrease-aware Heterogeneous Graph Neural Networks (LDHGNN), comprising a loss-decrease-aware training schedule (LDTS) and a probabilistic sampling strategy that selects easier samples based on epoch-to-epoch loss decreases, controlled by a schedule $\lambda_t \in (0,1]$ with pacing functions. Empirical results on the ogbn-mag benchmark show LDHGNN achieving substantial accuracy gains (e.g., $0.8836$ valid, $0.8789$ test) over prior methods like CLGNN and RpHGNN, with an ~8 percentage-point improvement over CLGNN. The approach demonstrates that incorporating relative difficulty signals into curriculum learning can significantly enhance HGNN performance, and the authors provide public code for reproducibility.

Abstract

In recent years, heterogeneous graph neural networks (HGNNs) have achieved excellent performance in handling heterogeneous information networks (HINs). Curriculum learning is a machine learning strategy where training examples are presented to a model in a structured order, starting with easy examples and gradually increasing difficulty, aiming to improve learning efficiency and generalization. To better exploit the rich information in HINs, previous methods have started to explore the use of curriculum learning strategy to train HGNNs. Specifically, these works utilize the absolute value of the loss at each training epoch to evaluate the learning difficulty of each training sample. However, the relative loss, rather than the absolute value of loss, reveals the learning difficulty. Therefore, we propose a novel loss-decrease-aware training schedule (LDTS). LDTS uses the trend of loss decrease between each training epoch to better evaluating the difficulty of training samples, thereby enhancing the curriculum learning of HGNNs for downstream tasks. Additionally, we propose a sampling strategy to alleviate training imbalance issues. Our method further demonstrate the efficacy of curriculum learning in enhancing HGNNs capabilities. We call our method Loss-decrease-aware Heterogeneous Graph Neural Networks (LDHGNN). The code is public at https://github.com/wangyili00/LDHGNN.