TEMPURA: Temporal Event Masked Prediction and Understanding for Reasoning in Action

TL;DR

TEMPURA tackles the challenge of fine-grained temporal reasoning in long videos by introducing a two-stage training framework that first performs masked event prediction to infer missing causal events and explanations, and then learns dense, timestamped video segmentation. The approach is trained on VER, a large-scale dataset of 500K untrimmed videos (18K hours) with dense, timestamp-aligned event captions and reasoning, enabling effective supervision for both stages. Empirical results on temporal grounding (Charades-STA) and highlight detection (QVHighlights) show substantial improvements over strong baselines, with gains achieved without task-specific fine-tuning and without heavy temporal encoders. The work demonstrates that integrating causal reasoning with fine-grained temporal segmentation yields robust long-video understanding, and the VER dataset provides a valuable resource for training such temporally aware vision-language models.

Abstract

Understanding causal event relationships and achieving fine-grained temporal grounding in videos remain challenging for vision-language models. Existing methods either compress video tokens to reduce temporal resolution, or treat videos as unsegmented streams, which obscures fine-grained event boundaries and limits the modeling of causal dependencies. We propose TEMPURA (Temporal Event Masked Prediction and Understanding for Reasoning in Action), a two-stage training framework that enhances video temporal understanding. TEMPURA first applies masked event prediction reasoning to reconstruct missing events and generate step-by-step causal explanations from dense event annotations, drawing inspiration from effective infilling techniques. TEMPURA then learns to perform video segmentation and dense captioning to decompose videos into non-overlapping events with detailed, timestamp-aligned descriptions. We train TEMPURA on VER, a large-scale dataset curated by us that comprises 1M training instances and 500K videos with temporally aligned event descriptions and structured reasoning steps. Experiments on temporal grounding and highlight detection benchmarks demonstrate that TEMPURA outperforms strong baseline models, confirming that integrating causal reasoning with fine-grained temporal segmentation leads to improved video understanding.

Figures (12)

• Figure 1: Our model, TEMPURA, is trained using a two-stage process for video understanding. The model first infers event structures and causal relationships by filling in missing details and reasoning about event sequences (e.g., recognizing that shrimp must be battered before frying). Second, it is learned to partition video into non-overlapping events and describe them in details. To achieve TEMPURA, we propose a new large-scale dataset consisting of 500k videos with dense event captions.
• Figure 2: Overview of TEMPURA’s two-stage training pipeline. (a) Masked Event Prediction Reasoning: The model infers missing events by analyzing the masked video context, generating both a textual description and step-by-step causal explanations. (b) Video Event Segmentation and Temporal Dense Captioning: The model partitions an untrimmed video into non-overlapping events, each aligned with precise start/end timestamps and enriched with detailed captions, thereby reinforcing a structured understanding of temporal progressions.
• Figure 3: Structured Training Data for Masked Event Prediction and Dense Event Caption
• Figure 4: VER Data Pipeline: The pipeline begins by filtering and categorizing a large video pool. GPT-4o then generates event captions with start/end times, followed by a temporal coherence check that discards invalid events. For valid events, a subset is masked to form a fill-in-the-blank task, and GPT-4o infers the missing segments—ultimately creating a dataset for video temporal understanding.
• Figure 5: Our model can segment videos into more fine-grained events, capturing subtle transitions and short-duration activities. In contrast, the baseline model (QwenVL2.5) tends to generate coarser segments. This difference suggests that our approach is more adept at recognizing and differentiating fine-grained patterns within video sequences, leading to detailed and structured event representation.