Language-TPP: Integrating Temporal Point Processes with Language Models for Event Analysis

TL;DR

Language-TPP presents a unified framework that integrates Temporal Point Processes with Large Language Models by introducing a novel temporal byte-token encoding to represent time intervals within standard LLM tokenizers. Using a decoder-only Transformer backbone, it models sequences of events described by time, type, and natural language descriptions, enabling next-event time, type, intensity, and description generation. The method achieves state-of-the-art results on multiple TPP tasks and demonstrates superior quality in generated event descriptions, particularly on datasets with textual descriptions. This approach offers a practical pathway for jointly modeling temporal dynamics and textual multimodal event data, with implications for more expressive predictive analytics and explanation generation in real-world sequences.

Abstract

Temporal Point Processes (TPPs) have been widely used for event sequence modeling, but they often struggle to incorporate rich textual event descriptions effectively. Conversely, while Large Language Models (LLMs) have been shown remarkable capabilities in processing textual data, they lack mechanisms for handling temporal dynamics. To bridge this gap, we introduce Language-TPP, a unified framework that integrates TPPs with LLMs for enhanced event sequence modeling. Language-TPP introduces a novel temporal encoding mechanism that converts continuous time intervals into specialized byte-tokens, enabling seamless integration with standard LLM architectures. This approach allows Language-TPP to achieve state-of-the-art performance across multiple TPP tasks, including event time prediction, type prediction, and intensity estimation, on five datasets. Additionally, we demonstrate that incorporating temporal information significantly improves the quality of generated event descriptions.

Figures (5)

• Figure 1: Temporal tokenization procedure in Language-TPP. The diagram illustrates how event times are converted into temporal byte tokens for processing by the model.
• Figure 2: Language-TPP processes tokenized event information including event type, description, and time through the QwenLM decoder. The decoder autoregressively generates information about the next event through next-token prediction, while the event intensity is modeled using the hidden state corresponding to the last token.
• Figure 3: Illustration of the textual template used to convert an event sequence into the input for the language model. Prepended by the sequence prompt, the event template structures each event with its type, description, and timestamp.
• Figure 4: Goodness-of-fit comparison on real-world datasets. Results reported in TLL. Best results are marked with stars.
• Figure 5: Results on Amazon Review dataset. (a): The RMSE$(\downarrow)$ on event time prediction; (b): the ROUGE-L$(\uparrow)$ score on textual event description generation; (c): comparison of sentiment polarity distribution of generated event descriptions.