CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment

TL;DR

CAV-MAE Sync presents a temporally aligned, self-supervised audio-visual framework that improves fine-grained correspondence by treating audio as a sequence aligned to video frames and by disentangling contrastive and reconstruction objectives with global tokens and register tokens. The method achieves state-of-the-art zero-shot retrieval on AudioSet and VGGSound, and competitive performance on classification and sound-prompted segmentation across AudioSet, VGGSound, and ADE20K_Sound, using a simpler architecture than many competitors. Key contributions include the temporal granularity of audio, the separation of learning objectives via global tokens, and the use of registers to reduce semantic load on patch tokens, all validated by extensive ablations. The approach offers practical benefits for efficient cross-modal retrieval and localization, with robust performance under self-supervised pretraining on AudioSet2M. Overall, CAV-MAE Sync advances fine-grained audio-visual alignment with tangible improvements in retrieval, classification, and localization tasks.

Abstract

Recent advances in audio-visual learning have shown promising results in learning representations across modalities. However, most approaches rely on global audio representations that fail to capture fine-grained temporal correspondences with visual frames. Additionally, existing methods often struggle with conflicting optimization objectives when trying to jointly learn reconstruction and cross-modal alignment. In this work, we propose CAV-MAE Sync as a simple yet effective extension of the original CAV-MAE framework for self-supervised audio-visual learning. We address three key challenges: First, we tackle the granularity mismatch between modalities by treating audio as a temporal sequence aligned with video frames, rather than using global representations. Second, we resolve conflicting optimization goals by separating contrastive and reconstruction objectives through dedicated global tokens. Third, we improve spatial localization by introducing learnable register tokens that reduce semantic load on patch tokens. We evaluate the proposed approach on AudioSet, VGG Sound, and the ADE20K Sound dataset on zero-shot retrieval, classification and localization tasks demonstrating state-of-the-art performance and outperforming more complex architectures.

Figures (6)

• Figure 1: By representing audio with multiple finer-grained representations aligned with individual video frames, CAV-MAE Sync improves the precision of audio-visual alignment, in contrast to the original CAV-MAE, which uses a global audio representation that struggles with fine-grained temporal correspondence.
• Figure 2: Overview of our approach. Our model processes video frames and audio segments in parallel through separate encoders $E_a$ and $E_v$, with the audio encoder $E_a$ operating on finer temporal granularity to better align with visual frames. Both modalities interact through the Joint Layer $L$ and the Joint Decoder $D$ The model is trained with both reconstruction and contrastive objectives.
• Figure 3: Illustration of our downstream tasks: (1) Classification: using CLS token with $f_{cls}$ projection for video-level prediction, and (2) Retrieval: computing similarity matrix R between audio query $A_q$ and video candidates $V_t$ for cross-modal matching.
• Figure 4: Impact of audio segment length on model performance. Experiments show $3$-$4$ second segments achieve optimal results while reducing computational costs compared to standard 10-second segments.
• Figure 5: Sound-prompted segmentation results showing localization maps generated from audio prompts from VGGSound classes like "writing on blackboard", "roller coaster", and "airplane". The model highlights relevant image regions corresponding to the audio, demonstrating strong audio-visual alignment for clear objects while more complex, cluttered scenes remain challenging.