Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Kernelized Edge Attention: Addressing Semantic Attention Blurring in Temporal Graph Neural Networks

Govind Waghmare, Srini Rohan Gujulla Leel, Nikhil Tumbde, Sumedh B G, Sonia Gupta, Srikanta Bedathur

TL;DR

KEAT introduces Kernelized Edge Attention for Temporal Graphs to address semantic attention blurring by decoupling node semantics from edge-temporal dynamics. By modulating edge features with continuous-time kernels (Laplacian, RBF, or learnable MLP), KEAT biases attention toward temporally relevant interactions while remaining agnostic to time-encoding schemes and architectures. The method is lightweight, plug-and-play, and shows consistent performance gains on dynamic link prediction and node classification across datasets and backbones (notably TGN and DyGFormer), with theoretical support for reduced higher-order moment sensitivity and stabilized attention. These results yield more accurate, interpretable, and temporally aware message passing in TGNNs, with practical impact for fraud detection, recommendations, and other time-evolving graph tasks.

Abstract

Temporal Graph Neural Networks (TGNNs) aim to capture the evolving structure and timing of interactions in dynamic graphs. Although many models incorporate time through encodings or architectural design, they often compute attention over entangled node and edge representations, failing to reflect their distinct temporal behaviors. Node embeddings evolve slowly as they aggregate long-term structural context, while edge features reflect transient, timestamped interactions (e.g. messages, trades, or transactions). This mismatch results in semantic attention blurring, where attention weights cannot distinguish between slowly drifting node states and rapidly changing, information-rich edge interactions. As a result, models struggle to capture fine-grained temporal dependencies and provide limited transparency into how temporal relevance is computed. This paper introduces KEAT (Kernelized Edge Attention for Temporal Graphs), a novel attention formulation that modulates edge features using a family of continuous-time kernels, including Laplacian, RBF, and learnable MLP variant. KEAT preserves the distinct roles of nodes and edges, and integrates seamlessly with both Transformer-style (e.g., DyGFormer) and message-passing (e.g., TGN) architectures. It achieves up to 18% MRR improvement over the recent DyGFormer and 7% over TGN on link prediction tasks, enabling more accurate, interpretable and temporally aware message passing in TGNNs.

Kernelized Edge Attention: Addressing Semantic Attention Blurring in Temporal Graph Neural Networks

TL;DR

KEAT introduces Kernelized Edge Attention for Temporal Graphs to address semantic attention blurring by decoupling node semantics from edge-temporal dynamics. By modulating edge features with continuous-time kernels (Laplacian, RBF, or learnable MLP), KEAT biases attention toward temporally relevant interactions while remaining agnostic to time-encoding schemes and architectures. The method is lightweight, plug-and-play, and shows consistent performance gains on dynamic link prediction and node classification across datasets and backbones (notably TGN and DyGFormer), with theoretical support for reduced higher-order moment sensitivity and stabilized attention. These results yield more accurate, interpretable, and temporally aware message passing in TGNNs, with practical impact for fraud detection, recommendations, and other time-evolving graph tasks.

Abstract

Temporal Graph Neural Networks (TGNNs) aim to capture the evolving structure and timing of interactions in dynamic graphs. Although many models incorporate time through encodings or architectural design, they often compute attention over entangled node and edge representations, failing to reflect their distinct temporal behaviors. Node embeddings evolve slowly as they aggregate long-term structural context, while edge features reflect transient, timestamped interactions (e.g. messages, trades, or transactions). This mismatch results in semantic attention blurring, where attention weights cannot distinguish between slowly drifting node states and rapidly changing, information-rich edge interactions. As a result, models struggle to capture fine-grained temporal dependencies and provide limited transparency into how temporal relevance is computed. This paper introduces KEAT (Kernelized Edge Attention for Temporal Graphs), a novel attention formulation that modulates edge features using a family of continuous-time kernels, including Laplacian, RBF, and learnable MLP variant. KEAT preserves the distinct roles of nodes and edges, and integrates seamlessly with both Transformer-style (e.g., DyGFormer) and message-passing (e.g., TGN) architectures. It achieves up to 18% MRR improvement over the recent DyGFormer and 7% over TGN on link prediction tasks, enabling more accurate, interpretable and temporally aware message passing in TGNNs.
Paper Structure (63 sections, 3 theorems, 36 equations, 15 figures, 14 tables)

This paper contains 63 sections, 3 theorems, 36 equations, 15 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Temporal Graph Neural Networks.
  4. Attention Mechanisms in TGNNs.
  5. Modulated Attention Beyond TGNNs.
  6. Temporal Information Modeling.
  7. Modular and Extendable TGNN Designs.
  8. Methodology
  9. Problem Setting and Notation
  10. Transformer-style Temporal Message Passing
  11. Time Encodings in Temporal Graphs
  12. Limitations of Time Encodings.
  13. Design Criteria for Temporal Kernels
  14. Family of Temporal Kernels
  15. Method: KEAT – Kernelized Attention over Time
  16. ...and 48 more sections

Key Result

Theorem 1

Let $p(t)$ be a probability density function supported on $[0, \infty)$ such that $\mathbb{E}[t^n] < \infty$ and $\mathbb{E}[\psi(t) t^n] < \infty$ for all $n \geq 0$, where $\psi(t)$ is a non-negative, monotonically decreasing kernel function. Let $\phi(t) = \sum_{n=0}^{\infty} c_n t^n$ be an analy Then, the sequence $\{R_n\}_{n=0}^\infty$ is strictly decreasing and converges to zero: $\lim_{n \t

Figures (15)

  • Figure 1: Temporal mismatch in dynamic graphs. While user profiles (nodes) change gradually (e.g., credit behavior), interactions like transactions (edges) vary rapidly and irregularly. Attention mechanisms in existing TGNNs often entangle these signals, leading to semantic attention blurring.
  • Figure 2: Temporal attention on tgbl-wiki illustrating semantic attention blurring. We plot attention over ten neighbors across relative time $\Delta t$. (Left) Standard attention assigns uniform attention values across time for each neighbor, leading to semantic blurring. This shows invariance of attention to time encodings or $\Delta t$. (Middle) KEAT introduces time-aware modulation, emphasizing recent (smaller $\Delta t$ value) and contextually relevant edges. (Right) The difference highlights how KEAT corrects the blurred focus of standard attention. See experimental section for details.
  • Figure 3: Spectral density for tgbl-datasets illustrating nature of periodicity.
  • Figure 4: Decay of the ratio $R_n = \frac{\mathbb{E}[t^n e^{-a t}]}{\mathbb{E}[t^n]}$ for increasing moment order $n$, where $t \sim \text{Exp}(1)$. This demonstrates how higher-order moments are increasingly suppressed under exponential kernel weighting.
  • Figure 5: Comparison of temporal kernels used in KEAT for edge modulation. Laplacian and RBF apply fixed exponential decays, while the MLP-based kernel learns a flexible decay pattern from data, allowing richer temporal biasing.
  • ...and 10 more figures

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 2
  • Theorem 3: Variance Reduction via Temporal Kernel