Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Foundation Models for Temporal Point Processes to Accelerate Scientific Discovery

David Berghaus, Patrick Seifner, Kostadin Cvejoski, Ramses J. Sanchez

TL;DR

The paper tackles the transferability challenge in temporal point process modeling by proposing a Foundation Inference Model trained on diverse synthetic Hawkes-like sequences to learn a prior over MTPP dynamics and to estimate the conditional intensity $\hat{\lambda}(t,\kappa|\mathcal{H}_t)$ from a short context. Using a transformer-based encoder-decoder architecture, the model enables zero-shot inference on new datasets and can be fine-tuned for improved accuracy. Key contributions include the first FIM for Marked Temporal Point Processes, synthetic-data pretraining with a Hawkes prior (including excitatory/inhibitory interactions), and strong zero-shot performance with gains from fine-tuning. This approach promises to accelerate scientific discovery by enabling instant, accurate analysis of new event sequences without retraining.

Abstract

Many scientific fields, from medicine to seismology, rely on analyzing sequences of events over time to understand complex systems. Traditionally, machine learning models must be built and trained from scratch for each new dataset, which is a slow and costly process. We introduce a new approach: a single, powerful model that learns the underlying patterns of event data in context. We trained this "foundation model" on millions of simulated event sequences, teaching it a general-purpose understanding of how events can unfold. As a result, our model can analyze new scientific data instantly, without retraining, simply by looking at a few examples from the dataset. It can also be quickly fine-tuned for even higher accuracy. This approach makes sophisticated event analysis more accessible and accelerates the pace of scientific discovery.

On Foundation Models for Temporal Point Processes to Accelerate Scientific Discovery

TL;DR

The paper tackles the transferability challenge in temporal point process modeling by proposing a Foundation Inference Model trained on diverse synthetic Hawkes-like sequences to learn a prior over MTPP dynamics and to estimate the conditional intensity from a short context. Using a transformer-based encoder-decoder architecture, the model enables zero-shot inference on new datasets and can be fine-tuned for improved accuracy. Key contributions include the first FIM for Marked Temporal Point Processes, synthetic-data pretraining with a Hawkes prior (including excitatory/inhibitory interactions), and strong zero-shot performance with gains from fine-tuning. This approach promises to accelerate scientific discovery by enabling instant, accurate analysis of new event sequences without retraining.

Abstract

Many scientific fields, from medicine to seismology, rely on analyzing sequences of events over time to understand complex systems. Traditionally, machine learning models must be built and trained from scratch for each new dataset, which is a slow and costly process. We introduce a new approach: a single, powerful model that learns the underlying patterns of event data in context. We trained this "foundation model" on millions of simulated event sequences, teaching it a general-purpose understanding of how events can unfold. As a result, our model can analyze new scientific data instantly, without retraining, simply by looking at a few examples from the dataset. It can also be quickly fine-tuned for even higher accuracy. This approach makes sophisticated event analysis more accessible and accelerates the pace of scientific discovery.
Paper Structure (6 sections, 2 equations, 2 figures)

This paper contains 6 sections, 2 equations, 2 figures.

Table of Contents

  1. Introduction and Related Work
  2. In-Context TPP Inference
  3. Synthetic Dataset Generation
  4. Foundation Inference Model Architecture
  5. Demonstration of Capabilities: Preliminary Results
  6. Conclusion

Figures (2)

  • Figure 1: Schematic of FIM-PPfim_pp. A context of event sequences is encoded by a transformer encoder. The result is used as memory by a transformer decoder, which takes a specific event history$\mathcal{H}_t$ as a query. The final embedding, combined with a target mark$\kappa$, is projected to parameters that define the conditional intensity function$\hat{\lambda}(t, \kappa \mid \mathcal{H}_t)$.
  • Figure 2: Demonstration of FIM-PP's zero-shot inference capabilities on examples from three synthetic datasets: a synthetic Poisson process, a process with a constant base intensity, and one with a gamma-distributed base intensity. The zero-shot model effectively captures the underlying dynamics by leveraging its pretrained prior. The ground truth intensity is shown alongside the model's estimate, illustrating a close match even without dataset-specific training.