Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Grounding

TL;DR

This work targets weakly supervised temporal video grounding by introducing Gaussian Mixture Proposals (GMP) that can express diverse query-driven events through learnable centers, widths, and importances learned directly over temporal locations. A pull-push learning scheme encourages moderately coupled Gaussians within each proposal, combining pulling for locality and pushing for distinctness, while an importance-based reconstructor assigns weights to masks to optimize reconstruction of the query. The GMP framework, supported by an MC-transformer based mask-importance module and a multiloss objective, achieves state-of-the-art performance on ActivityNet Captions and Charades-STA, with extensive ablations validating each component. Overall, expressing temporal structure with a learnable Gaussian mixture and carefully balanced coupling improves grounding accuracy under weak supervision, offering a scalable alternative to sliding-window or single-Gaussian approaches.

Abstract

In the weakly supervised temporal video grounding study, previous methods use predetermined single Gaussian proposals which lack the ability to express diverse events described by the sentence query. To enhance the expression ability of a proposal, we propose a Gaussian mixture proposal (GMP) that can depict arbitrary shapes by learning importance, centroid, and range of every Gaussian in the mixture. In learning GMP, each Gaussian is not trained in a feature space but is implemented over a temporal location. Thus the conventional feature-based learning for Gaussian mixture model is not valid for our case. In our special setting, to learn moderately coupled Gaussian mixture capturing diverse events, we newly propose a pull-push learning scheme using pulling and pushing losses, each of which plays an opposite role to the other. The effects of components in our scheme are verified in-depth with extensive ablation studies and the overall scheme achieves state-of-the-art performance. Our code is available at https://github.com/sunoh-kim/pps.

Figures (5)

• Figure 1: Weakly supervised temporal video grounding. (a) Previous methods use sliding windows (left) or a single Gaussian proposal (right), which has a predetermined shape. (b) The proposed method generates a Gaussian mixture proposal trained to be moderately coupled with a pull-push learning scheme to capture diverse query-relevant events.
• Figure 2: The overall scheme of the proposed method. The Gaussian mixture proposal generator produces multiple Gaussian masks from the features representing both the video and sentence query. For the positive proposals, we define a Gaussian mixture proposal, where multiple Gaussian masks are combined via attentive pooling using the importance weights from the importance-based reconstructor. Further, to generate moderately coupled masks in the mixture proposal, we propose the pull-push learning scheme using $\mathcal{L}_{pull}$, $\mathcal{L}^{intra}_{push}$, and $\mathcal{L}^{inter}_{push}$. The importance-based reconstructor leverages the proposals to produce the reconstructed query from the hidden query.
• Figure 3: Ablation studies by varying the number of positive and negative proposals $K$ and the number of Gaussian masks of an easy negative proposal $E_{en}$.
• Figure 4: Ablation studies by varying $\alpha$ values for the pull-push learning scheme on the ActivityNet Captions dataset.
• Figure 5: Qualitative results on the Activity-Net Captions dataset. Given a video and a query, PPS yields a predicted temporal location (red). We also visualize the predictions of variants using a positive proposal of one Gaussian mask without the mixture (yellow) or excluding a pulling loss (green) or excluding pushing losses (purple).