Taming Modality Entanglement in Continual Audio-Visual Segmentation

TL;DR

The paper tackles the problem of fine-grained continual learning for audio-visual segmentation, introducing Continual Audio-Visual Segmentation (CAVS) and identifying two key modality-entanglement challenges: multi-modal semantic drift and co-occurrence confusion. It proposes Collision-based Multi-modal Rehearsal (CMR), comprising Multi-modal Sample Selection (MSS) to reinforce cross-modal alignment and Collision-based Sample Rehearsal (CSR) to bias rehearsal toward confusable old classes, with formal mechanisms to measure modality consistency and collision frequency. The authors instantiate AVSBench-derived incremental datasets (AVSBench-CI/CIS/CIM) and demonstrate that CMR achieves state-of-the-art performance across class-incremental AVS tasks, including ablations showing the effectiveness of MSS and CSR and their synergy. The work advances practical multi-modal continual learning by enabling more robust, pixel-level audio-visual segmentation under sequential task regimes, with strong evidence across CNN and Transformer backbones.

Abstract

Recently, significant progress has been made in multi-modal continual learning, aiming to learn new tasks sequentially in multi-modal settings while preserving performance on previously learned ones. However, existing methods mainly focus on coarse-grained tasks, with limitations in addressing modality entanglement in fine-grained continual learning settings. To bridge this gap, we introduce a novel Continual Audio-Visual Segmentation (CAVS) task, aiming to continuously segment new classes guided by audio. Through comprehensive analysis, two critical challenges are identified: 1) multi-modal semantic drift, where a sounding objects is labeled as background in sequential tasks; 2) co-occurrence confusion, where frequent co-occurring classes tend to be confused. In this work, a Collision-based Multi-modal Rehearsal (CMR) framework is designed to address these challenges. Specifically, for multi-modal semantic drift, a Multi-modal Sample Selection (MSS) strategy is proposed to select samples with high modal consistency for rehearsal. Meanwhile, for co-occurence confusion, a Collision-based Sample Rehearsal (CSR) mechanism is designed, allowing for the increase of rehearsal sample frequency of those confusable classes during training process. Moreover, we construct three audio-visual incremental scenarios to verify effectiveness of our method. Comprehensive experiments demonstrate that our method significantly outperforms single-modal continual learning methods.

Figures (11)

• Figure 1: Illustration of CAVS and two challenges. In the figure, three sequential tasks are presented from top to bottom. Gray boxes: learned or background classes, light-orange boxes: target classes to be learned. Multi-modal semantic drift occurs when a learned class (e.g., darkgreen drum) is labeled as background in task $t$, despite the presence of its corresponding sound in the audio. This drift causes the model to suffer catastrophic forgetting of the modality semantic associations specific to the drum. Co-occurrence confusion occurs when, in a previous task (e.g. task $t$), two classes frequently co-occur (guitar and woman). After learning a new task, the model tends to misclassify the old classes (guitar) as the new ones (woman).
• Figure 2: Overview of the proposed CMR framework. The CMR framework introduces a novel rehearsal-based method for continual audio-visual segmentation. Our method consists of two key modules: (a) Multi-modal Sample Selection (MSS) strategy for samples rehearsal, which identifies samples with high modality consistency by computing the difference in mean Intersection-over-Union ($mIoU$) between uni-modal and multi-modal models. (b) Collision-based Sample Rehearsal (CSR) strategy that dynamically adjusts the rehearsal frequency of samples based on the collision between the old model and current ground truth.
• Figure 3: Illustration of inter-modal consistent samples and entanglement of modality. (a) Cases 1 and 4 don't appear in practice because selection uses already well-trained samples where audio and video predictions match the ground truth. Case 3 represents samples characterized by multi-modal semantic drift and is typically excluded due to substantially large $|\Delta(S_{a})|$. Conversely, Case 2 is kept because of its cross-modal semantic consistency. (b) Classes with infrequent co-occurrence exhibit weak audio-visual entanglement, while frequent co-occurrence leads to strong cross-modal entanglement (e.g., guitar sounds and images of women).
• Figure 4: Illustration of the collision-based sample rehearsal: for each sample, we calculate conflicts between old model predictions (dog) and current ground truth (baby). Aggregating these across all samples yields the collision frequency $\mathcal{F}$, quantifying confusion between old and new classes. By aligning the distribution of replayed samples with the collision frequency, the model is better guided to disentangle incorrect modality semantic associations during training.
• Figure 5: $mIoU$ on the AVSBench-CIS and AVSBench-CIM datasets for different class-incremental audio-visual segmentation scenarios. The red line represents the upper bound. The upper section compares different methods and our method under different incremental settings on AVSBench-CIS, including both disjoint and overlapped scenarios. The lower section provides a similar comparison for AVSBench-CIM, showcasing the performance of our method.