T2SGrid: Temporal-to-Spatial Gridification for Video Temporal Grounding

TL;DR

The core idea of T2SGrid is to process video content in clips rather than individual frames, which enables the use of composite text timestamps to establish global temporal awareness and achieves superior performance on standard VTG benchmarks.

Abstract

Video Temporal Grounding (VTG) aims to localize the video segment that corresponds to a natural language query, which requires a comprehensive understanding of complex temporal dynamics. Existing Vision-LMMs typically perceive temporal dynamics via positional encoding, text-based timestamps, or visual frame numbering. However, these approaches exhibit notable limitations: assigning each frame a text-based timestamp token introduces additional computational overhead and leads to sparsity in visual attention, positional encoding struggles to capture absolute temporal information, and visual frame numbering often compromises spatial detail. To address these issues, we propose Temporal to Spatial Gridification (T2SGrid), a novel framework that reformulates video temporal understanding as a spatial understanding task. The core idea of T2SGrid is to process video content in clips rather than individual frames. we employ a overlapping sliding windows mechanism to segment the video into temporal clips. Within each window, frames are arranged chronologically in a row-major order into a composite grid image, effectively transforming temporal sequences into structured 2D layouts. The gridification not only encodes temporal information but also enhances local attention within each grid. Furthermore, T2SGrid enables the use of composite text timestamps to establish global temporal awareness. Experiments on standard VTG benchmarks demonstrate that T2SGrid achieves superior performance.

Figures (5)

• Figure 1: Comparison between T2SGrid and previous Vision-LLMs. (a) Traditional methods process frames sequentially or apply temporal pooling to capture information at multiple scales. However, sequential processing and pooling can cause information loss, obscure local temporal details within a window, and often require additional time encoding/label for temporal modeling. (b) Our T2SGrid method folds multiple frames within a temporal window into the spatial dimension via gridification, allowing direct processing a temporal window by the standard 2D ViT. This leverages the model's strong spatial reasoning capability for temporal understanding.
• Figure 2: Illustration of Qwen2-VL temporal reasoning on 2D grid layouts. The model correctly infers the temporal order of events (before and after) and accurately identifies the biting action in frame 6.
• Figure 3: Overview of our T2SGrid framework. The original video frames are first arranged into a 2D grid in a row-major order (gridication) to enable spatialized temporal reasoning. A lightweight composite textual timestamp is incorporated to provide global temporal awareness. Our framework can operate in a training-free manner or be further enhanced via LoRA fine-tuning.
• Figure 4: Top: spatial and temporal attention with sequential frame input. Bottom: spatial and temporal attention with grid-based input. The grid method captures dynamic actions and maintain focus on the correct temporal intervals.
• Figure 5: Qualitative Comparison with State-of-the-Art. Our method outperforms TRACE guo2024trace and Numpro numberit on Charades by accurately identifying event boundaries in challenging scenes.