Learning Audio-Video Modalities from Image Captions

TL;DR

The paper tackles the scarcity of large-scale video-language data by mining millions of weakly labeled video-text pairs from image-caption datasets. It transfers captions from Conceptual Captions 3M to frames in online videos to create VideoCC3M, enabling an audiovisual transformer to achieve competitive text-video retrieval and video captioning with 20x less data than HowTo100M and state-of-the-art results in text-audio retrieval. The proposed model uses a dual-stream audiovisual encoder with a shared projection space of dimension $D=256$ trained via a contrastive $NCE$ objective, and extends to a GPT-2–style decoder for captioning. This approach demonstrates strong zero-shot generalization and suggests scalable, cost-efficient pretraining for multimodal video-language understanding, with potential societal benefits and biases to be considered in deployment.

Abstract

A major challenge in text-video and text-audio retrieval is the lack of large-scale training data. This is unlike image-captioning, where datasets are in the order of millions of samples. To close this gap we propose a new video mining pipeline which involves transferring captions from image captioning datasets to video clips with no additional manual effort. Using this pipeline, we create a new large-scale, weakly labelled audio-video captioning dataset consisting of millions of paired clips and captions. We show that training a multimodal transformed based model on this data achieves competitive performance on video retrieval and video captioning, matching or even outperforming HowTo100M pretraining with 20x fewer clips. We also show that our mined clips are suitable for text-audio pretraining, and achieve state of the art results for the task of audio retrieval.

Figures (11)

• Figure 1: Mining Audio-video clips automatically. We use the images in image captioning datasets as 'seed' frames to mine related audio-visual clips. For each seed image-caption pair in a dataset, we find frames in videos with high similarity scores to the seed image. We then extract short video clips around the matching frames and transfer the caption to those clips. This gives us free captioning supervision for video and audio clips.
• Figure 2: Examples of clips with captions that are mined automatically. For each seed image, we show 3 'matched' clips obtained using our automatic video mining method. For the first 2 clips, we show only a single frame, but for the third clip we present 2 frames to show motion, either of the subjects in the video (first 3 rows) or small camera motion (last 2 rows). Note the diversity in the mined clips, for example the different pitching poses and angles (first row) and the different types of statues (fourth row). Clips in the second row also contain audio relevant to the caption. Note frames may have been cropped and resized for ease of visualisation. More results are provided in the appendix.
• Figure 3: Domains in VideoCC3M vs HowTo100M. VideoCC3M has a more diverse and balanced range of domains, 'Other' here includes a variety of content such as music videos, sports, politics, vlogs and so on. Note how almost half of HowTo100M videos are food-related (cooking videos). More details are provided in the appendix.
• Figure 4: Examples from VideoCC3M of automatically mined clips with relevant audio to the caption. We show a single relevant frame from each clip as a proxy for visualising the audio. The accompanying audio contains (left to right) the sounds of a baby gurgling, music and water flowing sounds. The left image is intentionally blurred.
• Figure 5: Effect of match threshold $\tau$ on mining statistics (left) and zero-shot performance on MSR-VTT (right). Increasing the threshold beyond 0.6 decreases the size of the dataset, which leads to a corresponding performance drop on zero-shot retrieval. We use an optimal match threshold of $0.6$.