Scanning Only Once: An End-to-end Framework for Fast Temporal Grounding in Long Videos

TL;DR

Temporal grounding in long videos is challenged by inefficiency and information loss from sliding-window methods. SOONet introduces an anchor-based, end-to-end framework that scans a long-form video only once, using multi-scale anchors and a coarse-to-fine strategy that combines inter-anchor context with intra-anchor content and boundary regression. Training employs a dual-form ApproxNDCG-based cross-modal alignment loss and IoU-based boundary regression loss to robustly align queries with anchors. Experiments on MAD and Ego4d demonstrate substantial gains in both grounding accuracy (Recall metrics at various IoU thresholds) and pipeline efficiency (order-of-magnitude speedups), highlighting the approach's practicality for long-form video understanding.

Abstract

Video temporal grounding aims to pinpoint a video segment that matches the query description. Despite the recent advance in short-form videos (\textit{e.g.}, in minutes), temporal grounding in long videos (\textit{e.g.}, in hours) is still at its early stage. To address this challenge, a common practice is to employ a sliding window, yet can be inefficient and inflexible due to the limited number of frames within the window. In this work, we propose an end-to-end framework for fast temporal grounding, which is able to model an hours-long video with \textbf{one-time} network execution. Our pipeline is formulated in a coarse-to-fine manner, where we first extract context knowledge from non-overlapped video clips (\textit{i.e.}, anchors), and then supplement the anchors that highly response to the query with detailed content knowledge. Besides the remarkably high pipeline efficiency, another advantage of our approach is the capability of capturing long-range temporal correlation, thanks to modeling the entire video as a whole, and hence facilitates more accurate grounding. Experimental results suggest that, on the long-form video datasets MAD and Ego4d, our method significantly outperforms state-of-the-arts, and achieves \textbf{14.6$\times$} / \textbf{102.8$\times$} higher efficiency respectively. Project can be found at \url{https://github.com/afcedf/SOONet.git}.

Figures (8)

• Figure 1: Pipeline comparison between sliding window-based methods (top) zhang2020learningzeng2020densezhang2020spansoldan2021vlg and our SOONet (bottom). It is noteworthy that the sliding window pipeline requires repeated inference on overlapped clips and the final result aggregation, while ours can deliver the result with one-time network execution. Detailed discussion can be found in \ref{['sec:efficiency']}.
• Figure 2: Overall architecture of our algorithm. The whole framework consists of three modules: the pre-ranking module aims to obtain coarse anchor rank by modeling inter-anchor context; the re-ranking module aims to obtain content-enhanced anchor rank by supplementing anchors with detailed content; the regression module aims to adjust anchor boundaries.
• Figure 3: Ablation study on the base anchor length, $C_{0}$.
• Figure 4: Ablation study on the temperature value, $\alpha$.
• Figure 5: Qualitative analysis on the re-ranking module with full-length anchor matching scores, where re-ranking helps localize the moment of interest more precisely.