OmniGround: A Comprehensive Spatio-Temporal Grounding Benchmark for Real-World Complex Scenarios

TL;DR

This work introduces OmniGround, a large and diverse spatio-temporal grounding benchmark with 3,475 videos across 81 categories to better reflect real-world complexity in STVG. It presents the Forward-Backward-Refinement (FBR) annotation pipeline and the DeepSTG evaluation framework, which together provide high-quality labels and diagnostic metrics (CMA, FCI, VSBI, NEI) to reveal limitations of existing models. The authors propose PG-TAF, a training-free two-stage framework that separates high-level temporal grounding from fine-grained spatio-temporal propagation, achieving strong gains on OmniGround ($m_{tIoU}=49.2\%$, $m_{vIoU}=36.2\%$) and outperforming end-to-end baselines. The work highlights significant gaps in current STVG methods when facing uncommon objects, similar-target discrimination, and linguistically complex queries, and provides a practical pathway to improve real-world performance through diagnostic benchmarks and robust, training-free baselines.

Abstract

Spatio-Temporal Video Grounding (STVG) aims to localize target objects in videos based on natural language descriptions. Despite recent advances in Multimodal Large Language Models, a significant gap remains between current models and real-world demands involving diverse objects and complex queries. We attribute this to limited benchmark scope, causing models to exhibit category bias, oversimplified reasoning, and poor linguistic robustness. To address these limitations, we introduce OmniGround, a comprehensive benchmark with 3,475 videos spanning 81 categories and complex real-world queries. We propose the Forward-Backward-Refinement annotation pipeline that combines multi-directional tracking with intelligent error correction for high-quality labels. We further introduce DeepSTG, a systematic evaluation framework quantifying dataset quality across four complementary dimensions beyond superficial statistics. Evaluations reveal performance average drop of 10.4% on complex real-world scenes, particularly with small/occluded objects and intricate spatial relations. Motivated by these, we propose PG-TAF, a training-free two-stage framework decomposing STVG into high-level temporal grounding and fine-grained spatio-temporal propagation. Experiments demonstrate PG-TAF achieves 25.6% and 35.6% improvements in m\_tIoU and m\_vIoU on OmniGround with consistent gains across four benchmarks.

Figures (8)

• Figure 1: Some examples of STVG MLLMs facing real-world complex scenarios (left: uncommon objects, middle: multiple similar target objects, right: queries with deep syntactic complexity). Content in green and red represents correct and wrong answers, respectively.
• Figure 2: Performance of STVG MLLMs on complex real-world inputs. Inconsistent trends indicate current datasets are insufficient for comprehensive real-world evaluation.
• Figure 3: Statistics of our OmniGround benchmark: (a) Video durations range from short clips to 140 seconds. (b) Predicates combine spatial and action elements, reflecting complex object interactions and motion. (c) Balanced category distribution. (d) 81 object categories.
• Figure 4: (a) Overview of our Forward-Backward-Refinement pipeline. (b) An illustration of external data augmentation and validation.
• Figure 5: The mainframe of PG-TAF. We decouple STVG into high-level temporal grounding and fine-grained spatio-temporal propagation.