ECLIPSE: Efficient Long-range Video Retrieval using Sight and Sound

TL;DR

EclipSE addresses the challenge of long-range text-to-video retrieval by replacing parts of dense video processing with dense audio cues and integrating a unified audiovisual transformer into CLIP. The approach enables bi-directional cross-modal attention between video and audio, allowing sparse frames to be enriched by audio dynamics, resulting in substantial efficiency gains (about 2.9x faster and 2.3x memory-saving) while achieving state-of-the-art accuracy on ActivityNet Captions, QVHighlights, YouCook2, DiDeMo, and Charades. Extensive ablations show the superiority of dual-path audiovisual attention across all layers, robustness to audio encoders and durations, and the critical impact of CLIP pretraining. Qualitative analyses illustrate improved handling of audio-centric queries and implicit sound localization, underscoring the practical value of audiovisual fusion for long-range retrieval tasks and informing future multimodal video understanding directions.

Abstract

We introduce an audiovisual method for long-range text-to-video retrieval. Unlike previous approaches designed for short video retrieval (e.g., 5-15 seconds in duration), our approach aims to retrieve minute-long videos that capture complex human actions. One challenge of standard video-only approaches is the large computational cost associated with processing hundreds of densely extracted frames from such long videos. To address this issue, we propose to replace parts of the video with compact audio cues that succinctly summarize dynamic audio events and are cheap to process. Our method, named ECLIPSE (Efficient CLIP with Sound Encoding), adapts the popular CLIP model to an audiovisual video setting, by adding a unified audiovisual transformer block that captures complementary cues from the video and audio streams. In addition to being 2.92x faster and 2.34x memory-efficient than long-range video-only approaches, our method also achieves better text-to-video retrieval accuracy on several diverse long-range video datasets such as ActivityNet, QVHighlights, YouCook2, DiDeMo and Charades.

Figures (8)

• Figure 1: Comparison of different high-level frameworks for long-range text-to-video retrieval. Most traditional text-to-video retrieval methods (Leftmost Column) are designed for short videos (e.g., 5-15 seconds in duration). Adapting these approaches to several-minute long videos by stacking more input frames (Middle Column) is impractical due to excessive computational cost. Instead, our proposed framework operates on sparsely sampled video frames and dense audio cues, which are cheaper to process (Rightmost Column). In addition to being more efficient, our framework also achieves higher text-to-video retrieval accuracy than standard video-only approaches.
• Figure 2: Our audiovisual framework scales to long videos more efficiently than dense video-only approaches.
• Figure 3: We compare EclipSE with CLIP4Clip with a varying number of frames. Our method outperforms CLIP4Clip while using the same number or even fewer frames.
• Figure 4: (a) In the left subfigure, we study different audiovisual block design. Joint AV refers to standard self-attention applied to concatenated audio and video tokens. A2V refers to a single cross-modal audio-to-video attention block (Eq. \ref{['eq:a2v_att']}). Lastly, A2V+V2A depicts our dual-pathway attention block design (Eq. \ref{['eq:a2v_att']} and Eq. \ref{['eq:v2a_att']}). Based on these results, we observe that dual-pathway attention achieves the best performance. For efficiency, we use $8$ frame inputs for these experiments. (b) In the middle subfigure, we also investigate different audio encoders applied to different duration audio segments. These results indicate that (i) longer audio typically improves the performance, (ii) EclipSE is robust to different audio encoders. (c) In the right subfigure, we study video retrieval accuracy as a function of the number of audiovisual attention blocks. Based on these results, we observe that injecting our proposed audiovisual attention block into every layer of our 12-layer EclipSE model leads to the best performance.
• Figure 5: Here, we illustrate our qualitative retrieval results on ActivityNet Captions iccv17_activitynet. We compare our audiovisual EclipSE model with a video-only CLIP4Clip arxiv_clip4clip. For a given a textual query (depicted in a green block), we visualize each method's top-1 retrieved video. Our results indicate that the video-only CLIP4Clip struggles with retrieval when textual queries include audio event descriptions, e.g., "a woman speaking to the camera", "a person playing the violin," etc. (see bolded text). In these cases, CLIP4Clip fails to retrieve the correct video instances, whereas EclipSE effectively leverages audiovisual cues for a successful retrieval.