Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Hawkes Processes to Attention: Time-Modulated Mechanisms for Event Sequences

Xinzi Tan, Kejian Zhang, Junhan Yu, Doudou Zhou

TL;DR

The paper addresses the challenge of modeling irregular, multi-type event sequences by deriving Hawkes Attention, a time-modulated self-attention operator grounded in multivariate Hawkes process theory. It replaces fixed temporal encodings with learnable per-type neural influence kernels and a low-rank influence matrix, enabling joint learning of timing and content interactions in a scalable Transformer-like architecture. Key contributions include a principled decomposition of Hawkes intensities, per-type kernels with multiplicative time modulation, and end-to-end training that yields improved next-event time and type predictions along with interpretable influence patterns. The method also generalizes to regular time series, offering a unified framework for asynchronous event streams and continuous signals with competitive performance across diverse domains and strong interpretability of temporal dynamics.

Abstract

Marked Temporal Point Processes (MTPPs) arise naturally in medical, social, commercial, and financial domains. However, existing Transformer-based methods mostly inject temporal information only via positional encodings, relying on shared or parametric decay structures, which limits their ability to capture heterogeneous and type-specific temporal effects. Inspired by this observation, we derive a novel attention operator called Hawkes Attention from the multivariate Hawkes process theory for MTPP, using learnable per-type neural kernels to modulate query, key and value projections, thereby replacing the corresponding parts in the traditional attention. Benefited from the design, Hawkes Attention unifies event timing and content interaction, learning both the time-relevant behavior and type-specific excitation patterns from the data. The experimental results show that our method achieves better performance compared to the baselines. In addition to the general MTPP, our attention mechanism can also be easily applied to specific temporal structures, such as time series forecasting.

From Hawkes Processes to Attention: Time-Modulated Mechanisms for Event Sequences

TL;DR

The paper addresses the challenge of modeling irregular, multi-type event sequences by deriving Hawkes Attention, a time-modulated self-attention operator grounded in multivariate Hawkes process theory. It replaces fixed temporal encodings with learnable per-type neural influence kernels and a low-rank influence matrix, enabling joint learning of timing and content interactions in a scalable Transformer-like architecture. Key contributions include a principled decomposition of Hawkes intensities, per-type kernels with multiplicative time modulation, and end-to-end training that yields improved next-event time and type predictions along with interpretable influence patterns. The method also generalizes to regular time series, offering a unified framework for asynchronous event streams and continuous signals with competitive performance across diverse domains and strong interpretability of temporal dynamics.

Abstract

Marked Temporal Point Processes (MTPPs) arise naturally in medical, social, commercial, and financial domains. However, existing Transformer-based methods mostly inject temporal information only via positional encodings, relying on shared or parametric decay structures, which limits their ability to capture heterogeneous and type-specific temporal effects. Inspired by this observation, we derive a novel attention operator called Hawkes Attention from the multivariate Hawkes process theory for MTPP, using learnable per-type neural kernels to modulate query, key and value projections, thereby replacing the corresponding parts in the traditional attention. Benefited from the design, Hawkes Attention unifies event timing and content interaction, learning both the time-relevant behavior and type-specific excitation patterns from the data. The experimental results show that our method achieves better performance compared to the baselines. In addition to the general MTPP, our attention mechanism can also be easily applied to specific temporal structures, such as time series forecasting.
Paper Structure (34 sections, 16 equations, 2 figures, 10 tables)

This paper contains 34 sections, 16 equations, 2 figures, 10 tables.

Table of Contents

  1. INTRODUCTION
  2. FROM HAWKES PROCESS TO ATTENTION
  3. Hawkes process decomposition
  4. Per-type neural influence kernels
  5. Hawkes Attention: time-modulated self-attention
  6. Intensity and training
  7. EXPERIMENTS
  8. Datasets
  9. Metrics
  10. Baselines
  11. Experimental Setup and Training Details
  12. Next Event Time RMSE Comparison
  13. Next Event Type Error Rate Comparison
  14. Ablation Study
  15. Comparing against THP
  16. ...and 19 more sections

Figures (2)

  • Figure 1: The complete model architecture from input sequence through event embedding, stacked Transformer-encoder layers (Feed-Forward Network after Masked Hawkes Attention) to hidden representations.
  • Figure 2: Sample $\phi$ curves of different event types