Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization

TL;DR

Facing the challenge of localizing sound sources without labeled data, the paper leverages CLIP's multimodal priors by introducing an AudioTokenizer that maps audio into CLIP-compatible tokens and an audio-visual grounding framework. The method grounds sounding regions via an Audio-Visual Grounder and aligns audio-driven embedding $A$ with visual features using contrastive losses $\mathcal{L}_{ACL_I}$ and $\mathcal{L}_{ACL_F}$, with an optional caption-guided objective $\mathcal{L}_{ACL_C}$ for LLM-based guidance. An extension uses an LLM to distill object-aware scene understanding during training to further boost alignment. Experiments across five tasks—including single/multi-source localization, segmentation, and interactive localization—show state-of-the-art zero-shot performance and strong generalization, validating the approach's practicality in real-world settings.

Abstract

Large-scale vision-language models demonstrate strong multimodal alignment and generalization across diverse tasks. Among them, CLIP stands out as one of the most successful approaches. In this work, we extend the application of CLIP to sound source localization, proposing a self-supervised method operates without explicit text input. We introduce a framework that maps audios into tokens compatible with CLIP's text encoder, producing audio-driven embeddings. These embeddings are used to generate sounding region masks, from which visual features are extracted and aligned with the audio embeddings through a contrastive audio-visual correspondence objective. Our findings show that alignment knowledge of pre-trained multimodal foundation model enables our method to generate more complete and compact localization for sounding objects. We further propose an LLM-guided extension that distills object-aware audio-visual scene understanding into the model during training to enhance alignment. Extensive experiments across five diverse tasks demonstrate that our method, in all variants, outperforms state-of-the-art approaches and achieves strong generalization in zero-shot settings.

Figures (10)

• Figure 1: The proposed CLIP based sound source localization method.
• Figure 2: Our sound source localization framework with Audio-Visual Alignment. The proposed method takes audio-visual pairs, translating audio signals into CLIP-compatible tokens via the Audio Tokenizer module to generate audio-driven embedding, $\mathbf{A}$. This embedding highlights sounding regions within the Audio-Visual Grounder module. With the sounding area masks, the Audio-Visual Alignment module extracts audio-grounded visual features at both image-level ($\boldsymbol{v}^I$) and feature-level ($\boldsymbol{v}^F$). These visual features and audio feature are aligned via contrastive learning.
• Figure 3: Our extended framework with LLM-Guided Object-Aware Alignment. The proposed alternative method extends our audio-visual alignment framework by incorporating LLM guidance. Given audio-visual pairs, captions from each modality, $\mathbf{t}^a$ and $\mathbf{t}^i$, are generated and processed by an LLM to extract object-aware scene information. This is then encoded using CLIP’s text encoder, $\mathbf{c}$, and aligned with the audio-driven embedding $\mathbf{A}$, via contrastive learning, serving as an auxiliary objective. Since CLIP’s text and visual features are aligned, this guidance implicitly reinforces audio-visual correspondence.
• Figure 4: cIoU scores of SoundNet-Flickr samples.
• Figure 5: Qualitative sound source localization results on various datasets.