Enhancing Temporal Awareness in LLMs for Temporal Point Processes

TL;DR

This paper tackles the challenge of modeling continuous-time event sequences by integrating temporal dynamics with semantic reasoning in large language models. It introduces TPP-TAL, which combines Temporal Cross-Fusion (TCF) and Multi-Scale Temporal Bias Transformer (MTBT) to explicitly align time information with contextual semantics before LLM processing, rather than relying on simple time embeddings. The approach jointly optimizes TPP likelihood, event-type classification, and timestamp regression, and demonstrates consistent improvements in log-likelihood, accuracy, and RMSE across diverse datasets. The work advances practical continuous-time event modeling with time-aware semantic representations and presents a scalable path toward multimodal and real-time event reasoning in LLM-based systems.

Abstract

Temporal point processes (TPPs) are crucial for analyzing events over time and are widely used in fields such as finance, healthcare, and social systems. These processes are particularly valuable for understanding how events unfold over time, accounting for their irregularity and dependencies. Despite the success of large language models (LLMs) in sequence modeling, applying them to temporal point processes remains challenging. A key issue is that current methods struggle to effectively capture the complex interaction between temporal information and semantic context, which is vital for accurate event modeling. In this context, we introduce TPP-TAL (Temporal Point Processes with Enhanced Temporal Awareness in LLMs), a novel plug-and-play framework designed to enhance temporal reasoning within LLMs. Rather than using the conventional method of simply concatenating event time and type embeddings, TPP-TAL explicitly aligns temporal dynamics with contextual semantics before feeding this information into the LLM. This alignment allows the model to better perceive temporal dependencies and long-range interactions between events and their surrounding contexts. Through comprehensive experiments on several benchmark datasets, it is shown that TPP-TAL delivers substantial improvements in temporal likelihood estimation and event prediction accuracy, highlighting the importance of enhancing temporal awareness in LLMs for continuous-time event modeling. The code is made available at https://github.com/chenlilil/TPP-TAL

Figures (2)

• Figure 1: The TPP-TAL framework for continuous-time event modeling. Given an input event sequence, each event is encoded with both temporal and type embeddings, which are jointly processed through two plug-and-play modules: Temporal Cross-Fusion (TCF) and Multi-Scale Temporal Bias Transformer (MTBT). TCF performs fine-grained fusion between temporal and semantic features within individual events, while MTBT captures multi-scale temporal dependencies across events through per-head temporal biases. The fused representations are then fed into a pretrained LLM, which generates contextual hidden states for downstream tasks, including event type prediction, event time prediction, and intensity function estimation.
• Figure 2: Visualization of Attention Distribution for Different MTBT Variants on SOF Dataset