Sim-DETR: Unlock DETR for Temporal Sentence Grounding

TL;DR

Temporal sentence grounding with DETR-based methods faces two core issues: conflicts between similar target segments and a tension between global semantic matching and local boundary localization. Sim-DETR addresses these with two decoder-level tweaks: Query Grouping and Ranking (QGR) reduces inter- and intra-query conflicts by soft-spatial grouping and joint ranking, while Global-Local Bridging (GLB) introduces a query-to-frame alignment loss to couple global semantics with frame-level localization. Empirical results across QVHighlights, Charades-STA, and TACoS show consistent SOTA gains and faster convergence, with ablations confirming the effectiveness of QGR and GLB. The approach yields a simple yet powerful baseline that strengthens DETR’s applicability to temporal grounding and provides a practical path for future research in TSG.

Abstract

Temporal sentence grounding aims to identify exact moments in a video that correspond to a given textual query, typically addressed with detection transformer (DETR) solutions. However, we find that typical strategies designed to enhance DETR do not improve, and may even degrade, its performance in this task. We systematically analyze and identify the root causes of this abnormal behavior: (1) conflicts between queries from similar target moments and (2) internal query conflicts due to the tension between global semantics and local localization. Building on these insights, we propose a simple yet powerful baseline, Sim-DETR, which extends the standard DETR with two minor modifications in the decoder layers: (1) constraining self-attention between queries based on their semantic and positional overlap and (2) adding query-to-frame alignment to bridge the global and local contexts. Experiments demonstrate that Sim-DETR unlocks the full potential of DETR for temporal sentence grounding, offering a strong baseline for future research.

Figures (6)

• Figure 1: Illustration of (a) temporal sentence grounding task, (b) phenomenon of random matching across multiple layers, and (c) the challenge of achieving the balance between focus on boundary for span prediction and alignment with global semantics.
• Figure 2: The impact of the number of queries and decoder layers.
• Figure 3: Distribution of query similarity for intra-segment (blue) and inter-segment (red) pairs. Our method can effectively distinguish intra-segment and inter-segment queries, ensuring more stable query-segment associations and reducing conflicts in query assignments across different segments.
• Figure 5: Global matching score (defined in Line-290) vs. local localization score (defined in Line-293). Compared to CG-DETR, our method concentrates a query's attention significantly more within a single GT segment, rather than dispersing it across multiple segments, leading to improved local localization scores.
• Figure 6: Framework of the proposed Sim-DETR, featuring two simple yet powerful modifications: Query Grouping and Ranking to mitigate cross-query conflicts and Global-Local Bridging to address global-local conflicts within each query.