ArrowGEV: Grounding Events in Video via Learning the Arrow of Time

TL;DR

ArrowGEV tackles the limitation of forward-only event grounding by explicitly learning the arrow of time through reinforcement learning. It processes forward and backward videos, classifies events as time-sensitive or time-insensitive, and optimizes a grounding reward that combines localization accuracy with temporal directionality, using a GRPO-based policy refinement and a dynamic curriculum. Key contributions include a novel temporal directionality reward, a principled event categorization, and extensive experiments showing improved grounding accuracy and out-of-distribution generalization across multiple GEV benchmarks and video understanding tasks. The approach advances robust video understanding by instilling temporal structure in VLM-based grounding, with practical implications for fine-grained video analysis and reasoning tasks.

Abstract

Grounding events in videos serves as a fundamental capability in video analysis. While Vision-Language Models (VLMs) are increasingly employed for this task, existing approaches predominantly train models to associate events with timestamps in the forward video only. This paradigm hinders VLMs from capturing the inherent temporal structure and directionality of events, thereby limiting robustness and generalization. To address this limitation, inspired by the arrow of time in physics, which characterizes the intrinsic directionality of temporal processes, we propose ArrowGEV, a reinforcement learning framework that explicitly models temporal directionality in events to improve both event grounding and temporal directionality understanding in VLMs. Specifically, we categorize events into time-sensitive (e.g., putting down a bag) and time-insensitive (e.g., holding a towel in the left hand). The former denote events whose reversal substantially alters their meaning, while the latter remain semantically unchanged under reversal. For time-sensitive events, ArrowGEV introduces a reward that encourages VLMs to discriminate between forward and backward videos, whereas for time-insensitive events, it enforces consistent grounding across both directions. Extensive experiments demonstrate that ArrowGEV not only improves grounding precision and temporal directionality recognition, but also enhances general video understanding and reasoning ability.

Figures (7)

• Figure 1: Two examples of Qwen2.5-VL-7B predicting event timestamps in forward and backward videos. In the top row, reversing the video changes the event semantics, while in the bottom row the event remains invariant. The model partially localizes events in forward videos but fails to recognize event absence in the first reversed case and cannot robustly localize the event in the second reversed video.
• Figure 2: Quatitative Analysis of Qwen2.5-VL-7B on GEV Benchmarks. Left: statistics of the portion of time-sensitive and time-insensitive events across three benchmarks. Right: R1@m metrics on the time-sensitive subsets of three benchmarks.
• Figure 3: Overview of ArrowGEV. First, we input the event and both forward and backward videos into the VLM to obtain the predicted timestamps in both directions. Then we calculate the reward based on the category of the event. Having the reward for samples, we use GRPO with difficulty-aware training strategies to optimize the VLM for improved localization accuracy and directionality understanding.
• Figure 4: OOD results on six general video understanding and reasoning benchmarks.
• Figure 5: Results of the temporal directionality discrepancy (TDD) metric on three benchmarks. The upper row reports results on the time-sensitive subset, where higher values indicate better temporal directionality understanding, while the bottom row shows results on the time-insensitive subset, where lower values are preferable.