Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation

TL;DR

The paper tackles audio-visual segmentation by addressing the need for robust cross-modal alignment and reducing biases in AVS benchmarks. It introduces the Visual Post-production (VPO) benchmark to create cost-effective, unbiased multi-object audio-visual data, and the Contrastive Audio-Visual Pairing (CAVP) framework to mine informative cross-modal samples via supervised contrastive learning. Empirical results on AVSBench and the new VPO benchmarks show state-of-the-art segmentation performance, validating both the benchmark design and the CAVP learning objective. This work offers a scalable dataset strategy and a principled learning method that improve generalization to diverse AVS conditions and across multi-object scenarios.

Abstract

Audio-visual segmentation (AVS) is a challenging task that involves accurately segmenting sounding objects based on audio-visual cues. The effectiveness of audio-visual learning critically depends on achieving accurate cross-modal alignment between sound and visual objects. Successful audio-visual learning requires two essential components: 1) a challenging dataset with high-quality pixel-level multi-class annotated images associated with audio files, and 2) a model that can establish strong links between audio information and its corresponding visual object. However, these requirements are only partially addressed by current methods, with training sets containing biased audio-visual data, and models that generalise poorly beyond this biased training set. In this work, we propose a new cost-effective strategy to build challenging and relatively unbiased high-quality audio-visual segmentation benchmarks. We also propose a new informative sample mining method for audio-visual supervised contrastive learning to leverage discriminative contrastive samples to enforce cross-modal understanding. We show empirical results that demonstrate the effectiveness of our benchmark. Furthermore, experiments conducted on existing AVS datasets and on our new benchmark show that our method achieves state-of-the-art (SOTA) segmentation accuracy.

Figures (16)

• Figure 1: Current AVS datasets zhou2022audio tend to assume specific objects as consistent sound sources. Such a bias influences AVS methods, like TPAVI zhou2022audio (2nd row), to favour segmenting the presumed sound source, even when replacing the original audio with different sound types such as a person speaking (2nd column), bird chirping (3rd column), or background noise (4th row). Our paper proposes a new cost-effective strategy to build a relatively unbiased audio-visual segmentation benchmark and a supervised contrastive learning method that mines informative samples to better constrain the learning of audio-visual embeddings (last row).
• Figure 2: VPO Benchmarks. Using four classes, including "female", "cat", "dog", and "car", the AVSBench (SS) (1st frame) provides pixel-level multi-class annotations to the images containing a single sounding object. The proposed VPO benchmarks (2nd frame to 4th frame) pair a subset of the segmented objects in an image with relevant audio files to produce pixel-level multi-class annotations.
• Figure 3: Synthesising stereo sound for the VPO-MSMI setting.
• Figure 4: Illustration of our CAVP method for the "Dog" anchor. Starting with the audio-visual foreground anchor set $\mathcal{E}^{\text{fg}}_{\text{dog}}$, we create the positive and negative audio-visual features denoted by $\mathcal{P}^{\text{fg}}$ and $\mathcal{N}^{\text{fg}} = \mathcal{N}^{\text{fg}}_{\mathsf{hard}} \cup \mathcal{N}^{\text{fg}}_{\mathsf{easy}}$ respectively defined in Eq. \ref{['eq:positive_negative_sets']}. The CAVP loss in Eq. \ref{['eq:contrastive_loss_cavp']} pulls the anchor and positive audio-visual features closer while repelling the anchor and negative audio-visual features.
• Figure 5: Qualitative audio-visual segmentation results on AVSBench-Semantics zhou2023audio by TPAVI zhou2022audio, AVSegFormer gao2023avsegformer, and our CAVP, which can be compared with the ground truth (GT) of the first row.