SVAG-Bench: A Large-Scale Benchmark for Multi-Instance Spatio-temporal Video Action Grounding

TL;DR

This work introduces Spatio-temporal Video Action Grounding (SVAG), a task that jointly detects, tracks, and temporally localizes all objects performing actions described in natural language. To support SVAG, the authors present SVAG-Bench, a large-scale, action-centric dataset with dense annotations (688 videos, ~19,590 records, 903 verbs), along with SVAGFormer, a modular transformer baseline that fuses spatial and temporal grounding, and SVAGEval, a standardized evaluation toolkit. Empirical results show current models struggle in dense, long-form scenes, with spatial grounding quality acting as a primary bottleneck and temporal grounding benefiting from post-processing. Collectively, these contributions provide a foundation for advancing vision-language models that reason about fine-grained object-action interactions across extended video sequences, with practical impact on embodied AI, surveillance, and interactive systems.

Abstract

Understanding fine-grained actions and accurately localizing their corresponding actors in space and time are fundamental capabilities for advancing next-generation AI systems, including embodied agents, autonomous platforms, and human-AI interaction frameworks. Despite recent progress in video understanding, existing methods predominantly address either coarse-grained action recognition or generic object tracking, thereby overlooking the challenge of jointly detecting and tracking multiple objects according to their actions while grounding them temporally. To address this gap, we introduce Spatio-temporal Video Action Grounding (SVAG), a novel task that requires models to simultaneously detect, track, and temporally localize all referent objects in videos based on natural language descriptions of their actions. To support this task, we construct SVAG-Bench, a large-scale benchmark comprising 688 videos, 19,590 annotated records, and 903 unique verbs, covering a diverse range of objects, actions, and real-world scenes. We further propose SVAGFormer, a baseline framework that adapts state of the art vision language models for joint spatial and temporal grounding, and introduce SVAGEval, a standardized evaluation toolkit for fair and reproducible benchmarking. Empirical results show that existing models perform poorly on SVAG, particularly in dense or complex scenes, underscoring the need for more advanced reasoning over fine-grained object-action interactions in long videos.

Figures (6)

• Figure 1: Comparison of existing video grounding paradigms with our proposed Spatio-temporal Video Action Grounding (SVAG) task. (a) SVG: Spatial Video Grounding focuses only on spatial localization and lacks temporal reasoning. (b) STVG: Spatio-Temporal Video Grounding jointly localizes objects over time but cannot handle multiple interacting instances. (c) VTG: Video Temporal Grounding identifies temporal segments but misses spatial localization. (d) Ours (SVAG): Unifies temporal and spatial grounding to detect and track multiple referent objects performing the queried action across time.
• Figure 2: Statistics of SVAG-Bench. The majority of queries fall within the range of 6 to 10 words.
• Figure 3: Overview of the SVAGFormer pipeline for Spatio-temporal Video Action Grounding (SVAG). Given a natural language query (e.g., “A person is dancing in the open area”), the model first performs temporal grounding to localize the relevant video segment (Start: 2543 → End: 2782), followed by spatial grounding to identify the target person across frames.
• Figure 4: Flowchart for processing evaluation. Spatial and temporal evaluations are conducted separately on the OVIS, MOT17, and MOT20. The results are averaged and combined to form the final result. Threshold $\alpha$ controls the relative importance of detection and association accuracy in HOTA.
• Figure 5: A qualitative visualization example of a zebra performing a fine-grained action: tilting its head to the left across the grass. TempRMOT can localize the object, even with subtle action.