Query-centric Audio-Visual Cognition Network for Moment Retrieval, Segmentation and Step-Captioning
Yunbin Tu, Liang Li, Li Su, Qingming Huang
TL;DR
The paper addresses the problem of user-centric, hierarchical video understanding, focusing on moment retrieval, moment segmentation, and step-captioning within the HIREST framework. It introduces QUAG, a two-module architecture with modality-synergistic perception (MSP) and query-centric cognition (QC$^2$) to build a query-centric audio-visual representation by modeling global AV alignment, local cross-modal interactions, and query-guided filtration. Key contributions include a global-to-local AV fusion strategy and a deep-query filtration mechanism, achieving state-of-the-art results on HIREST and strong generalization to TVSum for query-based video summarization. The approach advances practical video understanding by aligning multi-modal content with user queries, enabling precise moment localization, structured segmentation, and captioning, with potential applicability to real-world video search and summarization tasks.
Abstract
Video has emerged as a favored multimedia format on the internet. To better gain video contents, a new topic HIREST is presented, including video retrieval, moment retrieval, moment segmentation, and step-captioning. The pioneering work chooses the pre-trained CLIP-based model for video retrieval, and leverages it as a feature extractor for other three challenging tasks solved in a multi-task learning paradigm. Nevertheless, this work struggles to learn the comprehensive cognition of user-preferred content, due to disregarding the hierarchies and association relations across modalities. In this paper, guided by the shallow-to-deep principle, we propose a query-centric audio-visual cognition (QUAG) network to construct a reliable multi-modal representation for moment retrieval, segmentation and step-captioning. Specifically, we first design the modality-synergistic perception to obtain rich audio-visual content, by modeling global contrastive alignment and local fine-grained interaction between visual and audio modalities. Then, we devise the query-centric cognition that uses the deep-level query to perform the temporal-channel filtration on the shallow-level audio-visual representation. This can cognize user-preferred content and thus attain a query-centric audio-visual representation for three tasks. Extensive experiments show QUAG achieves the SOTA results on HIREST. Further, we test QUAG on the query-based video summarization task and verify its good generalization.