Fostering Video Reasoning via Next-Event Prediction

TL;DR

This work introduces Next-Event Prediction (NEP), a self-supervised objective that trains multimodal LLMs to predict unseen future events from past video context, thereby improving temporal reasoning. NEP is instantiated with the V1-33K dataset and evaluated via the FutureBench benchmark, which tests coherence, causality, and multi-hop reasoning in future event trajectories. The authors explore multiple instruction-tuning strategies (SFT, CFT, Distill, Mix) and compare NEP against traditional video tasks, finding NEP enhances temporal understanding while preserving general video comprehension. They also study reinforcement learning with NEP, showing robust performance on FutureBench but with some trade-offs on general benchmarks, highlighting the need for balanced data, supervision, and potential biases in pursuit of temporal reasoning capabilities.

Abstract

Next-token prediction serves as the foundational learning task enabling reasoning in LLMs. But what should the learning task be when aiming to equip MLLMs with temporal reasoning capabilities over video inputs? Existing tasks such as video question answering often rely on annotations from humans or much stronger MLLMs, while video captioning tends to entangle temporal reasoning with spatial information. To address this gap, we propose next-event prediction (NEP), a learning task that harnesses future video segments as a rich, self-supervised signal to foster temporal reasoning. We segment each video into past and future frames: the MLLM takes the past frames as input and predicts a summary of events derived from the future frames, thereby encouraging the model to reason temporally in order to complete the task. To support this task, we curate V1-33K, a dataset comprising 33,000 automatically extracted video segments spanning diverse real-world scenarios. We further explore a range of video instruction-tuning strategies to study their effects on temporal reasoning. To evaluate progress, we introduce FutureBench to assess coherence in predicting unseen future events. Experiments validate that NEP offers a scalable and effective training paradigm for fostering temporal reasoning in MLLMs.

Figures (8)

• Figure 1: Comparison of Video Instruction Tuning tasks. (1) Video Q&A: Extracting answers from a single key frame; (2) Captioning: Summarizing from frame‐by‐frame visual perception of observed videos; (3) Next‐Event Prediction: Predicting the summary of future frames by visual perception of observed past frames and temporal reasoning with commonsense knowledge. As the example in the given first part video, after a defensive stop, the team may push fast in transition (knowledge)—but with under two minutes left in the fourth quarter (visual facts), a coach might call a timeout, or the players may slow the tempo to ensure careful execution.
• Figure 2: Reasoning structure underlying NEP. Each node is a potential event or action derived from visual cues, branching into alternative scenarios such as failing to defend or being pushed in transition. The red line highlights actual event sequence observed in the video. Comments provide reasoning for less likely scenarios.
• Figure 3: Distribution of data source and video length in V1-33K.The inner circle illustrates the distribution of data sources. The outer circle further segments each source according to video length categories. Only length categories comprising more than 4% of the dataset are labeled explicitly in the outer circle.
• Figure 4: Overview of the four-stage V1-33K construction pipeline: Fact Translation, Analysis, Segmentation, and Reasoning & Critique.
• Figure 5: Task demonstration of FutureBench. This figure presents two paradigms for future event prediction: Extrapolation and Interpolation. In the Extrapolation task (Top), the model observes the initial video (Current Event) and is required to sequentially predict a series of future events (Caption 1 → Caption 2 → Caption 3 → ...) leading up to the final event (Caption N). In the Interpolation task (Bottom), the model observes the initial video (Current Event) and is provided with the first future event (Caption 1), an anchor future event (Caption K), and the final event (Caption N) and must infer the most plausible intermediate events that bridge the temporal gap. Distractors involve Caption 0 of the current event to require the model to understand the given video. Questions and answer options above are simplified for clarity and brevity.