T-VSL: Text-Guided Visual Sound Source Localization in Mixtures

TL;DR

Localizing visual sound sources in multi-source video mixtures is difficult due to entangled audio-visual cues and lack of clean single-source references. The paper proposes T-VSL, a text-guided framework that leverages the tri-modal AudioCLIP embedding to disentangle per-source audio-visual correspondence by using text as coarse supervision, along with audio and visual conditioning blocks and an audio-visual correspondence module. It detects $K$ visual-sounding class instances, refines per-source features, and generates per-source heatmaps through cosine similarity in a shared embedding space. Across MUSIC, VGGSound, and VGGSound-Instruments, T-VSL achieves state-of-the-art results for both single- and multi-source localization and shows strong zero-shot transfer and robustness to higher source counts. The work highlights the value of cross-modal grounding with text guidance for robust, flexible sound localization in complex scenes, and releases code.

Abstract

Visual sound source localization poses a significant challenge in identifying the semantic region of each sounding source within a video. Existing self-supervised and weakly supervised source localization methods struggle to accurately distinguish the semantic regions of each sounding object, particularly in multi-source mixtures. These methods often rely on audio-visual correspondence as guidance, which can lead to substantial performance drops in complex multi-source localization scenarios. The lack of access to individual source sounds in multi-source mixtures during training exacerbates the difficulty of learning effective audio-visual correspondence for localization. To address this limitation, in this paper, we propose incorporating the text modality as an intermediate feature guide using tri-modal joint embedding models (e.g., AudioCLIP) to disentangle the semantic audio-visual source correspondence in multi-source mixtures. Our framework, dubbed T-VSL, begins by predicting the class of sounding entities in mixtures. Subsequently, the textual representation of each sounding source is employed as guidance to disentangle fine-grained audio-visual source correspondence from multi-source mixtures, leveraging the tri-modal AudioCLIP embedding. This approach enables our framework to handle a flexible number of sources and exhibits promising zero-shot transferability to unseen classes during test time. Extensive experiments conducted on the MUSIC, VGGSound, and VGGSound-Instruments datasets demonstrate significant performance improvements over state-of-the-art methods. Code is released at https://github.com/enyac-group/T-VSL/tree/main

Figures (4)

• Figure 1: Comparison of our T-VSL with state-of-the-art methods on single-source (Top Row) and multi-source (Bottom Row) sound localization on VGGSound Sources chen2021localizing, VGGSound-Instruments hu2022mix, and MUSIC zhao2018the benchmark datasets. We use the same setup for the fair comparison.
• Figure 2: The proposed text-guided visual sound source localization (T-VSL) framework (for $K = 2$). We use the text modality to disentangle the fine-grained audio-visual correspondence from mixtures. Initially, we detect the audio-visual class instances from multi-source mixtures using AudioCLIP joint embedding space. Later, categorical text features of each detected $K$ classes are used as coarse guidance in conditioning blocks to extract categorical visual and audio features. Afterwards, cross-modal feature alignment on extracted categorical features is performed in audio-visual correspondence block. Finally, cosine similarity of mean categorical audio features, and aligned visual features are used recursively to extract localization map of each class.
• Figure 3: We present qualitative comparisons on challenging multi-source localization with SOTA single and multi-source baseline methods. Here, blue color represents high-attention values to the sounding object, and red color represents low-attention values. The proposed T-VSL can selectively isolate the sounding regions from the background and generates more precise localization maps for sounding sources.
• Figure 4: We present additional qualitative comparisons on challenging multi-source localization with SOTA single and multi-source baseline methods. Here, blue color represents high-attention values to the sounding object, and red color represents low-attention values. Similar to our prior observation, the proposed T-VSL generates more precise localization maps for sounding sources by selectively isolating the sounding regions from the background.