Inference of Sequential Patterns for Neural Message Passing in Temporal Graphs

TL;DR

This work is the first to introduce statistically informed GNNs that leverage temporal and causal sequence anomalies, and represents a path for bridging the gap between statistical graph inference and neural graph representation learning, with potential applications to static GNNs.

Abstract

The modelling of temporal patterns in dynamic graphs is an important current research issue in the development of time-aware GNNs. Whether or not a specific sequence of events in a temporal graph constitutes a temporal pattern not only depends on the frequency of its occurrence. We consider whether it deviates from what is expected in a temporal graph where timestamps are randomly shuffled. While accounting for such a random baseline is important to model temporal patterns, it has mostly been ignored by current temporal graph neural networks. To address this issue we propose HYPA-DBGNN, a novel two-step approach that combines (i) the inference of anomalous sequential patterns in time series data on graphs based on a statistically principled null model, with (ii) a neural message passing approach that utilizes a higher-order De Bruijn graph whose edges capture overrepresented sequential patterns. Our method leverages hypergeometric graph ensembles to identify anomalous edges within both first- and higher-order De Bruijn graphs, which encode the temporal ordering of events. The model introduces an inductive bias that enhances model interpretability. We evaluate our approach for static node classification using benchmark datasets and a synthetic dataset that showcases its ability to incorporate the observed inductive bias regarding over- and under-represented temporal edges. We demonstrate the framework's effectiveness in detecting similar patterns within empirical datasets, resulting in superior performance compared to baseline methods in node classification tasks. To the best of our knowledge, our work is the first to introduce statistically informed GNNs that leverage temporal and causal sequence anomalies. HYPA-DBGNN represents a path for bridging the gap between statistical graph inference and neural graph representation learning, with potential applications to static GNNs.

Figures (11)

• Figure 1: Inference procedure leading to the dynamic graph used for neural message passing. (a) Example of sequence data adapted from LaRock_2020_hypa. (b) First- (blue) and higher-order (orange) De Bruijn graphs encoding temporal ordered time-stamped edges are compared to random graph ensemble null model with shuffled time-stamped k-1-order edges. (c) The graphs are corrected by introducing a statistical principled bias that revalues all edges ($w_{\langle AXC \rangle} \approx w_{\langle BXD \rangle} > w_{\langle BXC \rangle}$) and removes under-represented edges, i.e. edges that appear with a high probability less than expected ($\langle AXD \rangle$). (d) The multi-order graph neural network is trained respecting the inferred graphs.
• Figure 2: Distribution of average HYPA scores of incident edges. For each node $v_j$ the average HYPA score is determined with $\overline{HYPA}^{(k)}(v_j) = \frac{1}{|S(v_j)|} \sum_{(v_i, v_j) \in S(v_j)} HYPA^{(k)}(v_i,v_j)$ with the incident edges $S(v_j) = \{(v_i, v_j) \in E^{(k)}: v_i \in V^{(k)}\}$. The box plots show the distribution of theses scores with respect to node classes. The synthetic data set is Weighted Sampling.
• Figure 3: This figure presents the sampling procedure for the synthetic path data. It consists of five steps (left to right): (1) sampling of first-order nodes (uniform distribution) from a set with two classes (blue and orange); (2) combining the sampled nodes into second order nodes; (3) sampling out-connection candidates from the set of second-order nodes (e.g., $(\langle A,A \rangle,\,\cdot\,)$ highlighted in green). (4) sampling in-connections for every out-stub we sample a valid in-stub (e.g., from $(\langle A,A \rangle,\,\cdot\,)$: $(\,\cdot\,,\langle A,C \rangle)$ or $(\,\cdot\,,\langle A,A \rangle)$ -- highlighted in grey). Valid in-stubs whose nodes belong to the same group have a 5% increased probability of being sampled ($(\,\cdot\,,\langle A,A \rangle)$ gets the bonus while $(\,\cdot\,,\langle A,C \rangle)$ does not). (5) the edges are saved as paths ($\langle A, A, A \rangle$).
• Figure 5: This plot presents the absolute difference of the second-order edge frequencies of the Weighted and Unweighted data set. Due to the random sampling there are edges that have a higher frequency in on or the other data set. This trend increases with for edges that have more candidates in the urn. The edges that represent paths connecting nodes from the same class are mostly more often sampled and thus overrepresented in the Weighted data set. However, compared to the absolute frequencies in \ref{['fig:edge_stat']} the deviations are minor such that edges with low frequencies can be overrepresented. HYPA-DBGNN learns this pattern.
• Figure : (a) First-order edge statistics. They are equal for both the Weighted and the Unweighted data set. The differences are not observable by comparing them with the expected first-order edge statistics. The heterogeneous distribution is clearly visible.