Weakly Supervised Temporal Sentence Grounding via Positive Sample Mining

TL;DR

This work tackles weakly supervised temporal sentence grounding (WSTSG) by addressing the challenge of treating all non-anchor samples as negatives. It introduces Positive Sample Mining (PSM), which partitions the training set for each anchor into semantically similar and dissimilar samples using text-query similarity, and leverages them with a PSM-guided contrastive loss and a PSM-guided rank loss to exploit inter-sample correlations. The approach, built on transformer-based query and proposal encoders, achieves state-of-the-art or competitive results on Charades-STA, ActivityNet Captions, and NExT-GQA without adding inference cost, and shows robust gains across backbones. Extensive ablations validate the benefits of using similar samples and the two PSM losses, while also outlining limitations such as added training cost and challenges with highly complex queries that could be mitigated by sub-event decomposition in future work.

Abstract

The task of weakly supervised temporal sentence grounding (WSTSG) aims to detect temporal intervals corresponding to a language description from untrimmed videos with only video-level video-language correspondence. For an anchor sample, most existing approaches generate negative samples either from other videos or within the same video for contrastive learning. However, some training samples are highly similar to the anchor sample, directly regarding them as negative samples leads to difficulties for optimization and ignores the correlations between these similar samples and the anchor sample. To address this, we propose Positive Sample Mining (PSM), a novel framework that mines positive samples from the training set to provide more discriminative supervision. Specifically, for a given anchor sample, we partition the remaining training set into semantically similar and dissimilar subsets based on the similarity of their text queries. To effectively leverage these correlations, we introduce a PSM-guided contrastive loss to ensure that the anchor proposal is closer to similar samples and further from dissimilar ones. Additionally, we design a PSM-guided rank loss to ensure that similar samples are closer to the anchor proposal than to the negative intra-video proposal, aiming to distinguish the anchor proposal and the negative intra-video proposal. Experiments on the WSTSG and grounded VideoQA tasks demonstrate the effectiveness and superiority of our method.

Figures (6)

• Figure 1: a) Existing methods regard all samples in the remaining training set as negative and push them further from the anchor proposal, which ignores the diversity of the training set. b) Our method mines similar training samples for each anchor sample to leverage more discriminative supervision from other video samples. Two schemes are proposed. 1) A PSM-guided contrastive loss is proposed to distinguish between similar and dissimilar sample pairs. 2) A PSM-guided rank loss is proposed to improve the distinction between the anchor proposal and the negative intra-video proposal.
• Figure 2: Overview of our proposed method. First, we mine positive samples from the remaining training set to capture sophisticated semantic correlations in the sample space. Next, the proposal and query encoders are adopted to extract the global representations of proposals and queries. For similar and dissimilar query-video pairs, we utilize the positive proposal feature from the proposal encoder to compute the PSM-guided contrastive loss. Finally, PSM-guided contrastive loss is adopted to distinguish between similar and dissimilar queries/proposals for the anchor proposal. Furthermore, a PSM-guided rank loss improves the distinction between the anchor proposal and the negative intra-video proposal, based on the distance to similar samples (i.e., the PSM guided contrastive loss).
• Figure 3: The architecture of query and proposal encoder. The main network is adopted from CPL zheng2022weakly. We add mean-pooling to extract the global query feature and employ weighted sum to extract global positive and negative proposal features.
• Figure 4: Ablation study by varying $\gamma_1$ to $\gamma_6$ for positive sample mining on the Charades-STA dataset. When varying each $\gamma_i$, the remaining $\gamma$ values are fixed.
• Figure 5: The t-SNE results of the proposal spaces encoded by baseline PPS and our method (PPS + PSM) on the Charades-STA dataset. Two test samples are randomly selected. For each anchor sample, we present one similar and one dissimilar sample for visualization.