Deep Continuous-Time State-Space Models for Marked Event Sequences

TL;DR

We address the challenge of modeling marked event sequences by marrying latent linear Hawkes dynamics with deep state-space models to form a continuous-time, highly expressive MTPP. The proposed S2P2 architecture stacks latent linear Hawkes layers with nonlinearities and exploits diagonalization and zero-order hold discretization to enable parallel inference with logarithmic depth. Empirically, S2P2 achieves state-of-the-art per-event log-likelihood across eight real-world datasets, averaging a 33% improvement over strong baselines, while also delivering solid time and mark predictions and well-calibrated uncertainties. The approach combines strong inductive biases for continuous-time event dynamics with scalable computation, and is complemented by an open-source PyTorch implementation via EasyTPP to support broad adoption in real-world applications.

Abstract

Marked temporal point processes (MTPPs) model sequences of events occurring at irregular time intervals, with wide-ranging applications in fields such as healthcare, finance and social networks. We propose the state-space point process (S2P2) model, a novel and performant model that leverages techniques derived for modern deep state-space models (SSMs) to overcome limitations of existing MTPP models, while simultaneously imbuing strong inductive biases for continuous-time event sequences that other discrete sequence models (i.e., RNNs, transformers) do not capture. Inspired by the classical linear Hawkes processes, we propose an architecture that interleaves stochastic jump differential equations with nonlinearities to create a highly expressive intensity-based MTPP model, without the need for restrictive parametric assumptions for the intensity. Our approach enables efficient training and inference with a parallel scan, bringing linear complexity and sublinear scaling while retaining expressivity to MTPPs. Empirically, S2P2 achieves state-of-the-art predictive likelihoods across eight real-world datasets, delivering an average improvement of 33% over the best existing approaches.

Figures (15)

• Figure 1: (a) A schematic of our proposed state-space point process (S2P2), a deep stack of novel latent linear Hawkes (LLH) layers interleaved with nonlinear and normalization layers creating an expressive MTPP architecture. (b) Summary table of results we present in \ref{['sec:exp']}. We summarize ranks for six key metrics, ranking the average held-out test set performance across eight real-world datasets for five randomly seeded models; as well as a holistic composite rank, defined as the average of the ranks. Our S2P2 model outperforms all baselines by almost an entire rank, strongly indicating state-of-the-art performance and robustness across metrics and datasets.
• Figure 2: Schematic of our state-space point process. We depict the internals of a single LLH layer of the model in their continuous time form (left) and as discrete computations (right). Black arrows can be concurrently computed in logarithmic time, and gray arrows in constant time.
• Figure 3: Intensity estimates from trained models when conditioned on an empty sequence $\mathcal{H}_t=\emptyset$ for NHP mei2017neural and our S2P2. Dotted lines show the ground truth intensity for an inhomogeneous Poisson process. S2P2 accurately captures the background intensity.
• Figure 4: Results for synthetic experiments in \ref{['sec:synthetic']}.
• Figure 5: Median runtime of various models against increasing sequence lengths when conditioning on a sequence (\ref{['alg:dhlp:get_right_state_limit']}) and for likelihood evaluation (\ref{['alg:dhlp:ll']}) over 20 random seeds (variance negligible). S2P2 is faster across a wide range of sequence lengths. Crosses indicate where THP runs out of memory or IFTPP throws an error.