Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Models as Masked Audio-Video Learners

Elvis Nunez, Yanzi Jin, Mohammad Rastegari, Sachin Mehta, Maxwell Horton

TL;DR

This work investigates integrating diffusion models into MAViL for audio-visual pre-training to learn richer representations while improving training efficiency. The proposed DiffMAViL framework replaces learnable mask tokens with diffused patches, and introduces efficiency enhancements such as cross-attention in the video decoder, a curriculum-based masking schedule, and adaptive batch sizing. Empirically, DiffMAViL achieves a 32% reduction in pre-training FLOPS and an 18% reduction in wall-clock time without degrading downstream audio classification performance, and in some cases shows modest gains over MAViL on audio tasks. The results suggest diffusion can enhance high-frequency features and efficiency in multimodal pre-training, with practical implications for large-scale unsupervised audio-visual learning.

Abstract

Over the past several years, the synchronization between audio and visual signals has been leveraged to learn richer audio-visual representations. Aided by the large availability of unlabeled videos, many unsupervised training frameworks have demonstrated impressive results in various downstream audio and video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. MAViL couples contrastive learning with masked autoencoding to jointly reconstruct audio spectrograms and video frames by fusing information from both modalities. In this paper, we study the potential synergy between diffusion models and MAViL, seeking to derive mutual benefits from these two frameworks. The incorporation of diffusion into MAViL, combined with various training efficiency methodologies that include the utilization of a masking ratio curriculum and adaptive batch sizing, results in a notable 32% reduction in pre-training Floating-Point Operations (FLOPS) and an 18% decrease in pre-training wall clock time. Crucially, this enhanced efficiency does not compromise the model's performance in downstream audio-classification tasks when compared to MAViL's performance.

Diffusion Models as Masked Audio-Video Learners

TL;DR

This work investigates integrating diffusion models into MAViL for audio-visual pre-training to learn richer representations while improving training efficiency. The proposed DiffMAViL framework replaces learnable mask tokens with diffused patches, and introduces efficiency enhancements such as cross-attention in the video decoder, a curriculum-based masking schedule, and adaptive batch sizing. Empirically, DiffMAViL achieves a 32% reduction in pre-training FLOPS and an 18% reduction in wall-clock time without degrading downstream audio classification performance, and in some cases shows modest gains over MAViL on audio tasks. The results suggest diffusion can enhance high-frequency features and efficiency in multimodal pre-training, with practical implications for large-scale unsupervised audio-visual learning.

Abstract

Over the past several years, the synchronization between audio and visual signals has been leveraged to learn richer audio-visual representations. Aided by the large availability of unlabeled videos, many unsupervised training frameworks have demonstrated impressive results in various downstream audio and video tasks. Recently, Masked Audio-Video Learners (MAViL) has emerged as a state-of-the-art audio-video pre-training framework. MAViL couples contrastive learning with masked autoencoding to jointly reconstruct audio spectrograms and video frames by fusing information from both modalities. In this paper, we study the potential synergy between diffusion models and MAViL, seeking to derive mutual benefits from these two frameworks. The incorporation of diffusion into MAViL, combined with various training efficiency methodologies that include the utilization of a masking ratio curriculum and adaptive batch sizing, results in a notable 32% reduction in pre-training Floating-Point Operations (FLOPS) and an 18% decrease in pre-training wall clock time. Crucially, this enhanced efficiency does not compromise the model's performance in downstream audio-classification tasks when compared to MAViL's performance.
Paper Structure (19 sections, 1 figure, 6 tables)

This paper contains 19 sections, 1 figure, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. DiffMAViL
  4. Model.
  5. Training efficiency.
  6. Experiments
  7. Main Results
  8. Ablations
  9. Replacing Self-Attention with Cross-Attention.
  10. Curriculum-Based Masking.
  11. Adaptive Batch Size.
  12. Conclusion
  13. MAViL and DiffMAViL with Cross-Attention
  14. MAViL and DiffMAViL Background
  15. MAViL
  16. ...and 4 more sections

Figures (1)

  • Figure 1: DiffMAViL architecture. Similar to the audio-video encoder-decoder architecture of MAViL MAViL, our DiffMAViL architecture takes as input RGB video frames and audio spectrograms. The spectrogram and RGB frames are first randomly masked, and visible patches from each modality are encoded via their respective encoders. Masked patches are diffused and concatenated with the outputs of the audio-video fusion encoder, which are then fed through the audio and video decoders to obtain reconstructions of the input spectrogram and RGB frames.