Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes

TL;DR

This work defines Ref-AVS, a pixel-level object segmentation task where natural language expressions enriched with multimodal cues (audio, visual, and temporal) guide the segmentation in dynamic scenes. It introduces Ref-AVS Bench, a large-scale dataset with ~4,000 videos and ~20,000 expressions spanning real-world audio-visual interactions and an unseen test subset for zero-shot evaluation. The authors propose EEMC, an end-to-end framework that fuses multimodal cues via a temporal bi-modal transformer and cross-modal prompting to steer a Mask-Query segmentation decoder, achieving state-of-the-art results on seen and unseen subsets and robust null-reference handling. The work demonstrates the benefits of integrating audio, visual, and language cues for fine-grained segmentation, with potential impact on attention focusing and object editing in immersive audio-visual experiences. The dataset and method establish a foundation for future large-scale multimodal foundation models in realistic audio-visual scene understanding.

Abstract

Traditional reference segmentation tasks have predominantly focused on silent visual scenes, neglecting the integral role of multimodal perception and interaction in human experiences. In this work, we introduce a novel task called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment objects within the visual domain based on expressions containing multimodal cues. Such expressions are articulated in natural language forms but are enriched with multimodal cues, including audio and visual descriptions. To facilitate this research, we construct the first Ref-AVS benchmark, which provides pixel-level annotations for objects described in corresponding multimodal-cue expressions. To tackle the Ref-AVS task, we propose a new method that adequately utilizes multimodal cues to offer precise segmentation guidance. Finally, we conduct quantitative and qualitative experiments on three test subsets to compare our approach with existing methods from related tasks. The results demonstrate the effectiveness of our method, highlighting its capability to precisely segment objects using multimodal-cue expressions. Dataset is available at \href{https://gewu-lab.github.io/Ref-AVS}{https://gewu-lab.github.io/Ref-AVS}.

Figures (15)

• Figure 1: Comparison of the Ref-AVS task with other related tasks. Ref-AVS challenges machines to locate objects of interest in the visual space using multimodal cues, just like humans do in the real world.
• Figure 1: Data examples in our dataset Ref-AVS Bench. Use five time-steps for clarity.
• Figure 2: The illustration of our Ref-AVS benchmark. Note,for , the sound volume increases successively from silent to loud. Our benchmark is meticulously designed to encompass multimodal expressions from multiple dimensions. By combining various types of modality expressions, we achieve a dataset that exhibits great diversity.
• Figure 2: Illustration of the categories distribution for our Ref-AVS dataset, where seen and unseen categories are provided. The object classes were carefully chosen to ensure diversity and capture the distributional characteristics observed in real-world scenarios.
• Figure 3: The overall architecture of our method EEMC. We utilize cached memory to preserve temporal information, enabling the model to capture temporal mutations. Modality encoding provides a more clear context for the multimodal cue features $Q_m$, using the modality identify tokens $[aud]$ and $[vis]$. Lastly, we achieve efficient prompting of the visual foundation model by employing cross-attention, where mask queries $q$ serve as the target and multimodal cue features $Q_m$ act as the source.