Span-based Localizing Network for Natural Language Video Localization

TL;DR

The paper tackles natural language video localization (NLVL) by reframing it as a multimodal span-based QA problem, treating video as a passage and the target moment as the answer span. It introduces VSLBase, a standard span-based QA model, and VSLNet, which adds a query-guided highlighting (QGH) module to address video-text differences such as continuity and frame-level sensitivity. Across Charades-STA, ActivityNet Caption, and TACoS, VSLNet achieves state-of-the-art or strong results, demonstrating the viability of span-based QA for NLVL. The work highlights the potential of multimodal span-based QA as a promising direction for accurate and efficient NLVL.

Abstract

Given an untrimmed video and a text query, natural language video localization (NLVL) is to locate a matching span from the video that semantically corresponds to the query. Existing solutions formulate NLVL either as a ranking task and apply multimodal matching architecture, or as a regression task to directly regress the target video span. In this work, we address NLVL task with a span-based QA approach by treating the input video as text passage. We propose a video span localizing network (VSLNet), on top of the standard span-based QA framework, to address NLVL. The proposed VSLNet tackles the differences between NLVL and span-based QA through a simple yet effective query-guided highlighting (QGH) strategy. The QGH guides VSLNet to search for matching video span within a highlighted region. Through extensive experiments on three benchmark datasets, we show that the proposed VSLNet outperforms the state-of-the-art methods; and adopting span-based QA framework is a promising direction to solve NLVL.

Figures (10)

• Figure 1: An illustration of localizing a temporal moment in an untrimmed video by a given language query.
• Figure 2: An overview of the proposed architecture for NLVL. The feature extractor is fixed during training. Figure (a) depicts the adoption of standard span-based QA framework, i.e., VSLBase. Figure (b) shows the structure of VSLNet.
• Figure 3: An illustration of foreground and background of visual features. $\alpha$ is the ratio of foreground extension.
• Figure 4: The structure of Query-Guided Highlighting.
• Figure 5: Similarity scores, $\mathcal{S}$, between visual and language features in the context-query attention. $a^s/a^e$ denote the start/end boundaries of ground truth video moment, $\hat{a}^{s}/\hat{a}^{e}$ denote the start/end boundaries of predicted target moment.