Long-range Modeling and Processing of Multimodal Event Sequences

TL;DR

The paper introduces MM-TPP, a unified multimodal temporal point process that integrates time, type, text, and visual content to generate rich analyses of event sequences. It extends Language-TPP by adding a vision modality and an adaptive compression mechanism based on temporal similarity to mitigate long-context challenges in multimodal data. The authors propose a two-stage training regime on multimodal templates and task-specific prompts, and validate their approach on DanmakuTPP and a new TAXI-PRO dataset, showing superior predictive accuracy and enhanced open-ended text generation/QA capabilities. Key contributions include the MM-TPP framework, the adaptive sequence compression strategy, the TAXI-PRO dataset, and comprehensive empirical evidence of improved long-range reasoning and multimodal generation in TPPs.

Abstract

Temporal point processes (TPPs) have emerged as powerful tools for modeling asynchronous event sequences. While recent advances have extended TPPs to handle textual information, existing approaches are limited in their ability to generate rich, multimodal content and reason about event dynamics. A key challenge is that incorporating multimodal data dramatically increases sequence length, hindering the ability of attention-based models to generate coherent, long-form textual descriptions that require long-range understanding. In this paper, we propose a novel framework that extends LLM-based TPPs to the visual modality, positioning text generation as a core capability alongside time and type prediction. Our approach addresses the long-context problem through an adaptive sequence compression mechanism based on temporal similarity, which reduces sequence length while preserving essential patterns. We employ a two-stage paradigm of pre-training on compressed sequences followed by supervised fine-tuning for downstream tasks. Extensive experiments, including on the challenging DanmakuTPP-QA benchmark, demonstrate that our method outperforms state-of-the-art baselines in both predictive accuracy and the quality of its generated textual analyses.

Figures (5)

• Figure 1: Overview of the MM-TPP framework. (a) Model Architecture: The framework is built upon Qwen2.5-VL, where multimodal events—consisting of time, type, text, and image information—are tokenized into unified token sequences. Visual content is encoded by the vision encoder and fused with other modalities via the language model decoder to perform autoregressive next-event prediction. (b) Adaptive Long Sequence Compression: Events with similar time intervals ($|\tau_i - \tau_{i-1}| < \Delta$) are compressed into a special token. This strategy enables efficient processing of long event sequences while preserving temporal structure and key event information.
• Figure 2: Evaluation of the compression mechanism and key hyperparameters on the DanmakuTPP dataset. (a) PPL comparison on the test set. Our compressed model consistently maintains a lower PPL than the uncompressed version, indicating effective long-range modeling. (b, c) Ablation results for context length and similarity threshold ($\Delta$) on RMSE and ACC, showing optimal performance with a larger context and $\Delta=0.2$.
• Figure 3: Distribution of the absolute difference between adjacent inter-event intervals on the training set. Vertical lines indicate the tested thresholds $\Delta$.
• Figure 4: An illustration of a single event in the TAXI-PRO dataset. Each event comprises four key components: (1) a 224$\times$224 map image centered on the event's coordinates, (2) a generated textual description including landmark and trip details, (3) a categorical event type and (4) a timestamp.
• Figure 5: A complete example of an uncompressed event representation