CoVA: Text-Guided Composed Video Retrieval for Audio-Visual Content
Gyuwon Han, Young Kyun Jang, Chanho Eom
TL;DR
This work expands Composed Video Retrieval to account for auditory as well as visual changes by introducing the CoVA framework and the AV-Comp benchmark. It proposes AVT Compositional Fusion, a query-aware fusion mechanism that weights visual, audio, and four textual components to produce a unified representation, trained with a symmetric InfoNCE objective. Empirical results on AV-Comp show that incorporating audio with video and text yields significant retrieval improvements over unimodal and simple fusion baselines, establishing AVT as a strong baseline for audio-visual-text alignment. The AV-Comp dataset and AVT method provide a practical tool for evaluating and diagnosing multimodal representations in content retrieval tasks with cross-modal variations.
Abstract
Composed Video Retrieval (CoVR) aims to retrieve a target video from a large gallery using a reference video and a textual query specifying visual modifications. However, existing benchmarks consider only visual changes, ignoring videos that differ in audio despite visual similarity. To address this limitation, we introduce Composed retrieval for Video with its Audio CoVA, a new retrieval task that accounts for both visual and auditory variations. To support this, we construct AV-Comp, a benchmark consisting of video pairs with cross-modal changes and corresponding textual queries that describe the differences. We also propose AVT Compositional Fusion (AVT), which integrates video, audio, and text features by selectively aligning the query to the most relevant modality. AVT outperforms traditional unimodal fusion and serves as a strong baseline for CoVA. Examples from the proposed dataset, including both visual and auditory information, are available at https://perceptualai-lab.github.io/CoVA/.