Unsupervised Audio-Visual Segmentation with Modality Alignment

TL;DR

The Modality Correspondence Alignment Alignment (MoCA) framework is proposed, which seamlessly integrates off-the-shelf foundation models like DINO, SAM, and ImageBind and leverages existing knowledge within these models and optimizes their joint usage for multimodal associations.

Abstract

Audio-Visual Segmentation (AVS) aims to identify, at the pixel level, the object in a visual scene that produces a given sound. Current AVS methods rely on costly fine-grained annotations of mask-audio pairs, making them impractical for scalability. To address this, we introduce unsupervised AVS, eliminating the need for such expensive annotation. To tackle this more challenging problem, we propose an unsupervised learning method, named Modality Correspondence Alignment (MoCA), which seamlessly integrates off-the-shelf foundation models like DINO, SAM, and ImageBind. This approach leverages their knowledge complementarity and optimizes their joint usage for multi-modality association. Initially, we estimate positive and negative image pairs in the feature space. For pixel-level association, we introduce an audio-visual adapter and a novel pixel matching aggregation strategy within the image-level contrastive learning framework. This allows for a flexible connection between object appearance and audio signal at the pixel level, with tolerance to imaging variations such as translation and rotation. Extensive experiments on the AVSBench (single and multi-object splits) and AVSS datasets demonstrate that our MoCA outperforms strongly designed baseline methods and approaches supervised counterparts, particularly in complex scenarios with multiple auditory objects. Notably when comparing mIoU, MoCA achieves a substantial improvement over baselines in both the AVSBench (S4: +17.24%; MS3: +67.64%) and AVSS (+19.23%) audio-visual segmentation challenges.

Figures (9)

• Figure 1: Emergence of coarse and noisy audio-pixel association in ImageBind's girdhar2023imagebind multimodal feature space. (a) Raw audio waveform, (b) input RGB, (c) using frozen ImageBind features, (d) MoCA (ours) -- generating finer masks and particularly only in the presence of sounding objects (see $2^{nd}$ frame, MoCA generates no mask due to silent audio).
• Figure 2: Overview of our proposed MoCA: (Left) In training, we generate positive and negative images by utilizing DINO embeddings. The fusion of these images with the corresponding audio from the anchor image yields audio-enhanced image features. An efficient learning process by the proposed audio-visual adapter weights is facilitated through the establishment of a contrastive training objective, utilizing our pixel matching aggregation strategy. (Right) In inference, we extract audio-enhanced image features and employ $k$-means clustering to form clusters. Optionally, we enhance object boundaries by matching the clustered feature map with mask proposals from a pre-trained SAM model. PMA refers to pixel matching aggregation. All vision-audio (ViA) model weights are shared and frozen.
• Figure 3: Fusing frozen ViA encoders using AdaAV: We augment frozen visual-audio (ViA) encoders using a lightweight audio-visual adapter, AdaAV, which, given an input image frame ($I_t$) and corresponding audio ($A_t$), generates an audio-enhanced image feature ($f^a$).
• Figure 4: Generating mask proposals using a cascade of a pre-trained Open-world Object Detector (OWOD) maaz2022class and Segment anything model (SAM) kirillov2023segment. Mask proposals represent the existing visual objects in a given frame.
• Figure 5: Qualitative comparisons(left: S4, right: MS3): (a) RGB frame, (b) ground truth mask, (c) AVSBench zhou2022audio(supervised), (d) OWOD-BIND (baseline), (e) MoCA (ours). MoCA produces more precise segmentation of overlapping objects without utilizing any audio-visual masks during training.