Learning to Visually Localize Sound Sources from Mixtures without Prior Source Knowledge

TL;DR

This work tackles multi-sound source localization without requiring prior knowledge of the number of sources. It introduces an Iterative Object Identification (IOI) module and an Object Similarity-aware Clustering (OSC) loss to automatically identify and cluster sound-making regions from audio-visual mixtures. The method employs a two-stream network to derive a sound-associated map and iteratively refines object hypotheses, guided by OSC to merge regions of the same object while separating distinct objects. Experiments on MUSIC and VGGSound demonstrate significant gains for both single- and multi-source localization, showing strong practical potential for real-world scenarios with no prior source information.

Abstract

The goal of the multi-sound source localization task is to localize sound sources from the mixture individually. While recent multi-sound source localization methods have shown improved performance, they face challenges due to their reliance on prior information about the number of objects to be separated. In this paper, to overcome this limitation, we present a novel multi-sound source localization method that can perform localization without prior knowledge of the number of sound sources. To achieve this goal, we propose an iterative object identification (IOI) module, which can recognize sound-making objects in an iterative manner. After finding the regions of sound-making objects, we devise object similarity-aware clustering (OSC) loss to guide the IOI module to effectively combine regions of the same object but also distinguish between different objects and backgrounds. It enables our method to perform accurate localization of sound-making objects without any prior knowledge. Extensive experimental results on the MUSIC and VGGSound benchmarks show the significant performance improvements of the proposed method over the existing methods for both single and multi-source. Our code is available at: https://github.com/VisualAIKHU/NoPrior_MultiSSL

Figures (10)

• Figure 1: Conceptual comparison between (a) existing methods and (b) the proposed method. The existing methods require prior source knowledge of the number of sound-making objects. In contrast, our method can effectively localize multiple sound-making objects without the need for prior source knowledge.
• Figure 1: Visualization of training time per epoch of our model. After epoch 10 (red line), the training time converges to about 500 seconds/epoch.
• Figure 2: Network configuration of the proposed sound source localization framework. GAP denotes global average pooling.
• Figure 2: Additional visualization results for VGGSound-Duet test set (two objects). 'Object $k$' is identified by our model without any prior knowledge.
• Figure 3: Explanation of the proposed Object Similarity-Aware Clustering (OSC) loss ($N=2$ example).