Cross-modal Cognitive Consensus guided Audio-Visual Segmentation

TL;DR

This work addresses Audio-Visual Segmentation (AVS) by bridging the semantic gap between global audio semantics and local visual regions. It introduces C3N, which first derives a unified-modal label via Cross-modal Cognitive Consensus Inference Module (C3IM) using confidence scores and label embeddings, then guides local feature highlighting with a parameter-free Cognitive Consensus guided Attention Module (CCAM). The approach fuses multi-scale visual features and audio with cross-modal non-local interactions, followed by a segmentation head trained with BCE and Dice losses. Experiments on AVSBench demonstrate state-of-the-art performance under both the S4 and MS3 settings, with ablations validating the contributions of C3IM and CCAM and showing modest computational overhead. The method advances practical AVS by enabling explicit semantic alignment that improves localization and mask accuracy, with future work exploring inter-frame temporal context.

Abstract

Audio-Visual Segmentation (AVS) aims to extract the sounding object from a video frame, which is represented by a pixel-wise segmentation mask for application scenarios such as multi-modal video editing, augmented reality, and intelligent robot systems. The pioneering work conducts this task through dense feature-level audio-visual interaction, which ignores the dimension gap between different modalities. More specifically, the audio clip could only provide a Global semantic label in each sequence, but the video frame covers multiple semantic objects across different Local regions, which leads to mislocalization of the representationally similar but semantically different object. In this paper, we propose a Cross-modal Cognitive Consensus guided Network (C3N) to align the audio-visual semantics from the global dimension and progressively inject them into the local regions via an attention mechanism. Firstly, a Cross-modal Cognitive Consensus Inference Module (C3IM) is developed to extract a unified-modal label by integrating audio/visual classification confidence and similarities of modality-agnostic label embeddings. Then, we feed the unified-modal label back to the visual backbone as the explicit semantic-level guidance via a Cognitive Consensus guided Attention Module (CCAM), which highlights the local features corresponding to the interested object. Extensive experiments on the Single Sound Source Segmentation (S4) setting and Multiple Sound Source Segmentation (MS3) setting of the AVSBench dataset demonstrate the effectiveness of the proposed method, which achieves state-of-the-art performance. Code is available at https://github.com/ZhaofengSHI/AVS-C3N.

Figures (10)

• Figure 1: Illustration of the proposed method. We first obtain the audio and visual semantics, which are then mapped into a unified embedding space. Based on the audio-visual semantic similarities, we infer the cognitive consensus as the guidance for the final segmentation.
• Figure 2: The overview of C3N. Firstly, the audio clip $A$ and visual frames $V=\{{{I}_{t}}\}_{t=1}^{T}$ are converted into the audio feature ${{F}_{a}}$ and visual features ${{F}_{v}}=\{{{V}_{i}}\}_{i=1}^{4}$. Then, we utilize audio and visual classification confidence and the similarities between label embeddings to construct the confidence re-weighted matrix. Next, we get the unified-modal label and feed it into hierarchical layers of visual backbone to obtain class-activated weights, which guide highlighting the local feature elements through the Cognitive Consensus guided Attention Module (CCAM). Finally, a cross-modal feature fusion module composed of Atrous Spatial Pyramid Pooling (ASPP) modules and cross-modal Non-local blocks, and a segmentation head are adopted for the prediction.
• Figure 3: Schematic of the C3IM. We first utilize pre-trained heads to obtain the audio/visual classification confidence ${{C}^{A}}$, ${{C}^{V}}$ independently. Then, we calculate similarities between modality-specific labels to construct the semantic similarity matrix ${{M}_{sim}}$. Finally, multiplying ${{C}^{A}}$, ${{C}^{V}}$, and ${{M}_{sim}}$ to obtain the confidence re-weighted matrix ${{M}_{cof}}$ and infer the cognitive consensus-based unified-modal label.
• Figure 4: The implementation of the commonly used cross-modal attention mechanism (shown in panel (a)), channel-spatial attention mechanism (shown in panel (b)), and our Cognitive Consensus guided Attention Module (CCAM) (shown in panel (c)).
• Figure 5: Audio-Visual Segmentation examples of TPAVI zhou2022audio and our C3N on the val/test set of AVSBench dataset.