Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network

TL;DR

This work tackles Temporal Sentence Grounding (TSG) in a multi-pair setting by introducing MP-TSG and the Multi-Thread Knowledge Transfer Network (MKTN). MKTN jointly trains multiple video–query pairs through four modules: cross-sentence semantic mining, adaptive cross-modal matching with a dynamic threshold $\phi$, object-phrase prototype matching, and activity-sentence prototype alignment, all fused in a Bi-GRU-based grounding head with a multi-task loss ${\mathcal{L}} = {\mathcal{L}}_{CL} + \lambda {\mathcal{L}}_1 + \gamma {\mathcal{L}}_2 + \mu {\mathcal{L}}_3$, and refined by a float boundary predictor. The approach leverages self-supervised cross-modal contrast and prototype-level alignment to reduce modality gaps and share knowledge across pairs, achieving state-of-the-art performance on ActivityNet Captions, TACoS, and Charades-STA, while also enabling plug-and-play improvements for existing TSG models. Overall, MP-TSG offers a scalable, efficient framework for grounding diverse sentence queries in long videos by exploiting inter-pair semantics and robust cross-modal representations.

Abstract

Given some video-query pairs with untrimmed videos and sentence queries, temporal sentence grounding (TSG) aims to locate query-relevant segments in these videos. Although previous respectable TSG methods have achieved remarkable success, they train each video-query pair separately and ignore the relationship between different pairs. We observe that the similar video/query content not only helps the TSG model better understand and generalize the cross-modal representation but also assists the model in locating some complex video-query pairs. Previous methods follow a single-thread framework that cannot co-train different pairs and usually spends much time re-obtaining redundant knowledge, limiting their real-world applications. To this end, in this paper, we pose a brand-new setting: Multi-Pair TSG, which aims to co-train these pairs. In particular, we propose a novel video-query co-training approach, Multi-Thread Knowledge Transfer Network, to locate a variety of video-query pairs effectively and efficiently. Firstly, we mine the spatial and temporal semantics across different queries to cooperate with each other. To learn intra- and inter-modal representations simultaneously, we design a cross-modal contrast module to explore the semantic consistency by a self-supervised strategy. To fully align visual and textual representations between different pairs, we design a prototype alignment strategy to 1) match object prototypes and phrase prototypes for spatial alignment, and 2) align activity prototypes and sentence prototypes for temporal alignment. Finally, we develop an adaptive negative selection module to adaptively generate a threshold for cross-modal matching. Extensive experiments show the effectiveness and efficiency of our proposed method.

Figures (3)

• Figure 1: (a) Example of temporal sentence grounding (TSG). (b) Previous TSG models regard each video-query pair independently. (c) Our proposed model explores the semantic relationship between different pairs to reduce the modality gap.
• Figure 2: Overview of our proposed MKTN for the MP-TSG task. Given some video-query pairs (e.g., the first and second videos correspond to one and two queries respectively), we first utilize video and query encoders to extract corresponding features. Then, we feed these features into four carefully-designed modules to fully explore the semantic relationships between videos and queries. In the cross-sentence semantic mining module, we mine the query-to-query relationship based on the cross-modal memory. For the adaptive video-query matching module, we adaptively learn the cross-modal semantic consistency with video-to-query relationship by a dynamic threshold $\phi$ and a contrastive loss $\mathcal{L}_{CL}$. In the object-phrase prototype matching module, we align appearance representations across modalities based on appearance and phrase prototypes. Similarly, we integrate motion representations by aligning activity and sentence prototypes. Best viewed in color.
• Figure 3: Performance comparison with state-of-the-art methods on Charades-STA. (a) compares the effectiveness (R@1, IoU=0.5) and the efficiency (QPS), (b) shows that our method can serve as a plug-and-play module to enhance their efficiency, (c) is the qualitative results.