Diversifying Query: Region-Guided Transformer for Temporal Sentence Grounding

TL;DR

Temporal sentence grounding seeks moments in video that align with a language description. The proposed Region-Guided Transformer (RGTR) replaces freely learned queries with anchor-pair moment queries organized by static and dynamic anchors, initialized via clustering on ground-truth spans to inject explicit regional priors and reduce proposal overlap. An IoU-aware scoring head couples localization quality with classification confidence, guiding the ranking of proposals by both semantic relevance and boundary accuracy. Across QVHighlights, Charades-STA, and TACoS, RGTR achieves state-of-the-art results and demonstrates robustness to distribution shifts, illustrating the value of regional guidance and boundary-aware scoring for precise temporal localization.

Abstract

Temporal sentence grounding is a challenging task that aims to localize the moment spans relevant to a language description. Although recent DETR-based models have achieved notable progress by leveraging multiple learnable moment queries, they suffer from overlapped and redundant proposals, leading to inaccurate predictions. We attribute this limitation to the lack of task-related guidance for the learnable queries to serve a specific mode. Furthermore, the complex solution space generated by variable and open-vocabulary language descriptions complicates optimization, making it harder for learnable queries to distinguish each other adaptively. To tackle this limitation, we present a Region-Guided TRansformer (RGTR) for temporal sentence grounding, which diversifies moment queries to eliminate overlapped and redundant predictions. Instead of using learnable queries, RGTR adopts a set of anchor pairs as moment queries to introduce explicit regional guidance. Each anchor pair takes charge of moment prediction for a specific temporal region, which reduces the optimization difficulty and ensures the diversity of the final predictions. In addition, we design an IoU-aware scoring head to improve proposal quality. Extensive experiments demonstrate the effectiveness of RGTR, outperforming state-of-the-art methods on QVHighlights, Charades-STA and TACoS datasets. Codes are available at https://github.com/TensorsSun/RGTR

Figures (9)

• Figure 1: Visualization comparison of all moment predictions on QVHighlights val split, for the 3 representative moment queries in EaTR jang2023knowing, TR-DETR sun2024tr, CG-DETR moon2023correlation and RGTR (Ours). x-axis denotes the normalized moment span center coordinate, y-axis denotes the normalized moment span length. All queries in previous methods generate numerous overlapped proposals. For example, the second query tends to predict long moments near the middle of the videos (higher middle area), but the proposals of short moments (lower area) conflict with this purpose, leading to ineffective predictions. In contrast, the predicted region of each query in our RGTR is distinct and more concentrated.
• Figure 2: (a) Overview of the proposed RGTR architecture. Given a video and a text description, we first employ two frozen pre-trained models to extract visual and textual features. Subsequently, the cross-modal alignment encoder is constructed to align and fuse the visual and textual features effectively. Then, we design a region-guided decoder to introduce the regional guidance for decoding process through a set of explicitly initialized anchor pairs. Finally, the IoU-aware scoring head generates high-quality proposals by incorporating localization quality to enhance the classification confidence estimation. (b) The clustering centers with regional priors are obtained by adopting k-means algorithm on the distribution of all ground-truth moment spans.
• Figure 3: The structure of our proposed region-guided decoder with anchor pair (static anchor and dynamic anchor), where PE means positional encoding.
• Figure 4: Ablation study on number of moment queries $\mathcal{K}$.
• Figure 5: Correlation between scores and ground-truth IoUs.