Probabilistic Modeling for Sequences of Sets in Continuous-Time

TL;DR

This work tackles the problem of modeling continuous-time sequences where each event is a set of items, rather than a single mark. It introduces a general framework that couples intensity-based recurrent MTPPs with set-valued marks, accessible via two conditional set models: Dynamic Bernoulli and Dynamic DPPs, enabling tractable inference and flexible queries. By decoupling time from set structure and employing additive set embeddings, the approach achieves improved predictive performance and enables efficient probabilistic queries using targeted importance sampling, including hitting-time and A-before-B queries. Empirical results across four real-world datasets demonstrate the benefits of dynamic set modeling and the practicality of query-based model selection, with Dynamic Bernoulli offering a favorable trade-off between expressivity and computation.

Abstract

Neural marked temporal point processes have been a valuable addition to the existing toolbox of statistical parametric models for continuous-time event data. These models are useful for sequences where each event is associated with a single item (a single type of event or a "mark") -- but such models are not suited for the practical situation where each event is associated with a set of items. In this work, we develop a general framework for modeling set-valued data in continuous-time, compatible with any intensity-based recurrent neural point process model. In addition, we develop inference methods that can use such models to answer probabilistic queries such as "the probability of item $A$ being observed before item $B$," conditioned on sequence history. Computing exact answers for such queries is generally intractable for neural models due to both the continuous-time nature of the problem setting and the combinatorially-large space of potential outcomes for each event. To address this, we develop a class of importance sampling methods for querying with set-based sequences and demonstrate orders-of-magnitude improvements in efficiency over direct sampling via systematic experiments with four real-world datasets. We also illustrate how to use this framework to perform model selection using likelihoods that do not involve one-step-ahead prediction.

Figures (13)

• Figure 1: Example sequence of set-valued events, where the item space is $\{\Large\textcolor{dot_red}{$$\bullet}, \Large\textcolor{dot_blue}{$$\bullet}, \Large\textcolor{dot_green}{$$\bullet}, \Large\textcolor{dot_orange}{$$\bullet}\}$ and $\mathcal{H}$ refers to history.
• Figure 2: A comparison of (a) traditional neural MTPPs and(b) our proposed model for sequences of sets, where the differences are highlighted in the orange boxes. In (a) $x_i$ represents a single item, whereas in (b) $x_i$ is a set of items.
• Figure 3: Example sequences for hitting time queries.
• Figure 4: Relative efficiency for queries of the form $p(\text{hit}(A) \leq t \mid \mathcal{H})$ for two model variants. Blue and red dashed lines refer to the multiplicative runtime of importance sampling compared to naive sampling.
• Figure 5: Relative efficiency results for the query $p(\text{hit}(A) < \text{hit}(B), \text{hit}(A) \leq t \mid \mathcal{H})$ with the same format as \ref{['fig:hit_eff']}.