Towards Long-Form Spatio-Temporal Video Grounding

TL;DR

This paper proposes an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG, which significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG.

Abstract

In real scenarios, videos can span several minutes or even hours. However, existing research on spatio-temporal video grounding (STVG), given a textual query, mainly focuses on localizing targets in short videos of tens of seconds, typically less than one minute, which limits real-world applications. In this paper, we explore Long-Form STVG (LF-STVG), which aims to locate targets in long-term videos. Compared with short videos, long-term videos contain much longer temporal spans and more irrelevant information, making it difficult for existing STVG methods that process all frames at once. To address this challenge, we propose an AutoRegressive Transformer architecture for LF-STVG, termed ART-STVG. Unlike conventional STVG methods that require the entire video sequence to make predictions at once, ART-STVG treats the video as streaming input and processes frames sequentially, enabling efficient handling of long videos. To model spatio-temporal context, we design spatial and temporal memory banks and apply them to the decoders. Since memories from different moments are not always relevant to the current frame, we introduce simple yet effective memory selection strategies to provide more relevant information to the decoders, significantly improving performance. Furthermore, instead of parallel spatial and temporal localization, we propose a cascaded spatio-temporal design that connects the spatial decoder to the temporal decoder, allowing fine-grained spatial cues to assist complex temporal localization in long videos. Experiments on newly extended LF-STVG datasets show that ART-STVG significantly outperforms state-of-the-art methods, while achieving competitive performance on conventional short-form STVG.

Figures (10)

• Figure 1: Comparison of existing STVG approaches ( e.g., TubeDETRcgstvgSTCATcsdvlwasim2024videogroundinggu2025knowing) that see the entire video to make a full prediction at once in (a) and ART-STVG that processes frames one at a time and is more suitable for handling long videos in LF-STVG in (b).
• Figure 2: Comparison on the datasets with minute-level videos. Existing STVG methods severely suffer localization in long sequences, and ART-STVG significantly surpasses these models. Moreover, the longer the video is, the larger the improvement of ART-STVG over other methods is.
• Figure 3: Overview of ART-STVG, comprising a multimodal encoder and an autoregressive decoder with cascade connection for target localization frame by frame.
• Figure 4: Illustration of the architectures for memory-augmented spatial decoder block in (a) and temporal decoder block in (b).
• Figure 5: Comparison of attention maps of the spatial query without and with using selective spatial memory. The red box indicates the target object to be located. Through comparison, we observe the use of selective spatial memory helps the model focus more on the target regions, which benefits the final target localization.