TubeRMC: Tube-conditioned Reconstruction with Mutual Constraints for Weakly-supervised Spatio-Temporal Video Grounding

TL;DR

This work tackles weakly-supervised Spatio-Temporal Video Grounding (STVG) by introducing TubeRMC, a framework that learns text-grounded tubes without bounding-box or timestamp annotations. It leverages pre-trained visual grounding to obtain frame-level text-grounding and then applies tube-conditioned reconstruction across temporal, spatial, and spatio-temporal perspectives, guided by Gaussian attention masks. Mutual constraints between spatial and temporal proposals further refine tube quality, improving consistency and temporal accuracy. Empirical results on VidSTG and HCSTVG show state-of-the-art performance and strong robustness, highlighting the method's potential to reduce annotation costs while enhancing target identification and tracking in video-grounded language understanding.

Abstract

Spatio-Temporal Video Grounding (STVG) aims to localize a spatio-temporal tube that corresponds to a given language query in an untrimmed video. This is a challenging task since it involves complex vision-language understanding and spatiotemporal reasoning. Recent works have explored weakly-supervised setting in STVG to eliminate reliance on fine-grained annotations like bounding boxes or temporal stamps. However, they typically follow a simple late-fusion manner, which generates tubes independent of the text description, often resulting in failed target identification and inconsistent target tracking. To address this limitation, we propose a Tube-conditioned Reconstruction with Mutual Constraints (\textbf{TubeRMC}) framework that generates text-conditioned candidate tubes with pre-trained visual grounding models and further refine them via tube-conditioned reconstruction with spatio-temporal constraints. Specifically, we design three reconstruction strategies from temporal, spatial, and spatio-temporal perspectives to comprehensively capture rich tube-text correspondences. Each strategy is equipped with a Tube-conditioned Reconstructor, utilizing spatio-temporal tubes as condition to reconstruct the key clues in the query. We further introduce mutual constraints between spatial and temporal proposals to enhance their quality for reconstruction. TubeRMC outperforms existing methods on two public benchmarks VidSTG and HCSTVG. Further visualization shows that TubeRMC effectively mitigates both target identification errors and inconsistent tracking.

Figures (5)

• Figure 1: (a) Comparison with previous WSTVG methods.(b) Illustration of tube-conditioned reconstruction. The tube that matches the query descriptions can correctly reconstruct masked phrases. Moreover, the process can refine tubes predicted in detection. [M] means the masked phrase.
• Figure 2: (a) TubeRMC Overview. It selects the most relevant bounding box for the subject token in each frame while extracting image-level visual-linguistic features (left), and generates temporal intervals (1D), spatial bounding boxes (2D), and spatio-temporal tube proposals (3D) for reconstruction (right). (b) Tube-conditioned Reconstruction Learning masks out phrases containing dynamic, static, or holistic event information in the sentence, then reconstructs the masked phrases condition on 1D, 2D, and 3D proposals. It further introduces mutual constraints between 1D and 2D proposals, employing time-to-space and space-to-time constraints to enhance spatio-temporal consistency in proposal generation.
• Figure 3: The effect of the Spatio-Temporal Decoder.
• Figure 4: Architecture of Tube-conditioned Reconstructor: (a) Overview; (b) Tube-conditioned Encoder layer design.
• Figure 5: Visualization of spatio-temporal predictions for our method (pink), MDETR-Zero (yellow) and ground truth (green).