DDAVS: Disentangled Audio Semantics and Delayed Bidirectional Alignment for Audio-Visual Segmentation

TL;DR

DDAVS tackles the AVS problem by explicitly disentangling audio semantics through a prototype-grounded Audio Query Module and sharpening them with a waveform-level Contrastive Optimization Module, then fusing modalities via a delayed, bidirectional Audio-Visual Alignment Module. Key contributions include a prototype memory bank anchoring for stable semantic grounding, a contrastive objective to improve discriminability under multi-source mixtures, and a multi-stage dual cross-attention fusion that delays cross-modal interaction to leverage higher-level representations. Empirical results on AVSBench and VPO show state-of-the-art performance, particularly in challenging multi-source and semantic-source settings, with strong qualitative evidence of cleaner, more source-consistent masks. These advances enhance robust localization of sound-producing objects in real-world scenes and offer a scalable approach for open-domain, time-evolving audio-visual perception.

Abstract

Audio-Visual Segmentation (AVS) aims to localize sound-producing objects at the pixel level by jointly leveraging auditory and visual information. However, existing methods often suffer from multi-source entanglement and audio-visual misalignment, which lead to biases toward louder or larger objects while overlooking weaker, smaller, or co-occurring sources. To address these challenges, we propose DDAVS, a Disentangled Audio Semantics and Delayed Bidirectional Alignment framework. To mitigate multi-source entanglement, DDAVS employs learnable queries to extract audio semantics and anchor them within a structured semantic space derived from an audio prototype memory bank. This is further optimized through contrastive learning to enhance discriminability and robustness. To alleviate audio-visual misalignment, DDAVS introduces dual cross-attention with delayed modality interaction, improving the robustness of multimodal alignment. Extensive experiments on the AVS-Objects and VPO benchmarks demonstrate that DDAVS consistently outperforms existing approaches, exhibiting strong performance across single-source, multi-source, and multi-instance scenarios. These results validate the effectiveness and generalization ability of our framework under challenging real-world audio-visual segmentation conditions. Project page: https://trilarflagz.github.io/DDAVS-page/

Figures (13)

• Figure 1: Qualitative comparison of our DDAVS model and previous methods. DDAVS consistently outperforms previous approaches in challenging scenarios involving multiple classes, multiple sources, small or distant sound sources, and off-screen audio cues.
• Figure 2: For Audio Disentanglement, prior methods (a) use learned queries for semantics gao2024avsegformerli2024qdformerwang2024avesformer or (b) derive disentangled features from K-nearest classes liu2025dynamic. In contrast, our method (c) uses an audio prototype memory bank to ground audio queries, coupled with contrastive optimization to enhance their discriminability and robustness. For Audio-Visual Alignment, existing methods either (d) treat audio features as a fixed condition wang2024avesformerzhou2022audiozhou2024audio, or (e) apply gating mechanisms to scale audio features for dual cross-attention liu2025dynamic. Our method, however, (f) performs dual cross-attention with delayed modality interaction to improve multimodal alignment robustness.
• Figure 3: Overview of the DDAVS framework. (a) The Audio Query Module (AQM) encodes original and augmented waveforms into disentangled semantic queries anchored to a prototype memory bank. (b) The Contrastive Optimization Module (COM) enhances query robustness through contrastive learning, used only during training. (c) The Audio-Visual Alignment Module (AVAM) fuses audio queries with visual features via stacked alignment blocks, and a lightweight decoder outputs the sound-conditioned segmentation masks.
• Figure 5: Visualization of attention maps of audio-injected transformers blocks at different layers. It is observed that injecting audio features into blocks 3 and 4 bringing clearer instance-level attention, compared to the blurry pattern at earlier blocks.
• Figure 6: Effect of the number of audio queries. Performance on AVS-MS3 (top) and AVSS (bottom) as the number of audio queries $n$ varies, where our choice $n=5$ (ours) achieves the best overall results before larger $n$ leads to performance degradation due to redundant queries.