How Do Optical Flow and Textual Prompts Collaborate to Assist in Audio-Visual Semantic Segmentation?

TL;DR

This work tackles audio-visual semantic segmentation (AVSS) by introducing Stepping Stone Plus (SSP), a framework that couples a pre-mask stage driven by optical flow with two textual prompts and a visual-textual alignment module to enhance cross-modal understanding. The pre-mask leverages motion cues to highlight potential sound sources, while textual prompts address stationary sound-emitting objects, with VTA unifying visual and textual representations and a post-mask loss reinforcing dynamic, sound-related features. Empirical results on AVSBench datasets show SSP achieves state-of-the-art performance, surpassing both fusion-based and prompt-based baselines in mIoU and F-score. The approach offers robust segmentation across moving and stationary sound sources, with practical implications for more accurate scene understanding in multimodal systems.

Abstract

Audio-visual semantic segmentation (AVSS) represents an extension of the audio-visual segmentation (AVS) task, necessitating a semantic understanding of audio-visual scenes beyond merely identifying sound-emitting objects at the visual pixel level. Contrary to a previous methodology, by decomposing the AVSS task into two discrete subtasks by initially providing a prompted segmentation mask to facilitate subsequent semantic analysis, our approach innovates on this foundational strategy. We introduce a novel collaborative framework, \textit{S}tepping \textit{S}tone \textit{P}lus (SSP), which integrates optical flow and textual prompts to assist the segmentation process. In scenarios where sound sources frequently coexist with moving objects, our pre-mask technique leverages optical flow to capture motion dynamics, providing essential temporal context for precise segmentation. To address the challenge posed by stationary sound-emitting objects, such as alarm clocks, SSP incorporates two specific textual prompts: one identifies the category of the sound-emitting object, and the other provides a broader description of the scene. Additionally, we implement a visual-textual alignment module (VTA) to facilitate cross-modal integration, delivering more coherent and contextually relevant semantic interpretations. Our training regimen involves a post-mask technique aimed at compelling the model to learn the diagram of the optical flow. Experimental results demonstrate that SSP outperforms existing AVS methods, delivering efficient and precise segmentation results.

Figures (8)

• Figure 1: Network architecture comparison, red text is utilized to denote the innovative modules within the respective methods. Fusion-based methods in (a), three categories of prompt-based methods (object queries, mask, and textual prompt in (b.1), (b.2), and (b.3), respectively), and our proposed method in (c). SSP introduces optical flow as an extra prompt alongside the pre-masking technique to the original image to achieve better segmentation mask results, and the incorporation of two textual prompts compensating for optical flow. The alignment module delivers more robust feature maps for visual inputs.
• Figure 2: Optical flow carries information about sound-emitting objects, which can be either (a) the dynamic object itself (the man) or (b) indirectly related to stationary object (the guitar).
• Figure 3: The workflow of the SSP begins by incorporating optical flow $\text{O}^\mathcal{T}$ as a prompt. We then generate the optical flow mask, denoted as $\mathcal{M}_\text{O}$, to interact with the GT mask $\mathcal{M}_\text{GT}$ within the pre-mask technique. This process returns the output $\mathcal{M}_\text{Pre}$, which is then multiplied with the visual features to identify moving and sound-producing objects; however, scenarios exist the sound-producing but stationary objects (the gray region in $\mathcal{M}_\text{Pre}$). To account for this, we introduce two textual prompts: $\text{A}_1$, which offers a comprehensive understanding of the scene and serves as the foundation for $\text{A}_2$, and $\text{A}_2$, indicates potential stationary objects capable of producing sound. The VTA facilitates the alignment process across modalities. The two aligned features, $\text{Align}_1$ and $\text{Align}_2$, are integrated with feature embeddings from the last hidden state of the visual decoder. In our training objective, we employ an additional mask prediction loss, $\mathcal{L}'_\text{mask}$, alongside the post-mask technique illustrated in Fig. \ref{['loss']}. This approach establishes the ground truth and compels the model to learn features that are both dynamic and sound-related.
• Figure 4: Ground truth label preparation for the post-mask technique, by having the intersection between the optical flow mask $\mathcal{M}_\text{O}$ and the original ground truth mask $\mathcal{M}_\text{GT}$.
• Figure 5: Five values of $\lambda'_\text{mask}$ assigned to $\mathcal{L}'_\text{mask}$, evaluating the impact to the overall performance using the MS3 dataset.