Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Critical Assessment of Visual Sound Source Localization Models Including Negative Audio

Xavier Juanola, Gloria Haro, Magdalena Fuentes

TL;DR

The paper tackles Visual Sound Source Localization (VSSL) in realistic scenarios by introducing negative audio—silence, noise, and offscreen sounds—and showing that existing evaluations overemphasize on-screen sources and prior object information. It introduces extended test sets on the VGG-SS and IS3 datasets and new metrics (pIA, AUC_N, F_LOC, F_AUC) to assess negative-audio performance and global results, plus IoU-based comparisons between localization maps. A universal localization threshold is derived from the distribution of maximum audio-visual similarity, enabling in-the-wild evaluation without prior knowledge of object visibility or size. Empirical results reveal that several SOTA models lack discrimination between positive and negative audio, underscoring the need for more robust VSSL approaches suitable for real-world deployment.

Abstract

The task of Visual Sound Source Localization (VSSL) involves identifying the location of sound sources in visual scenes, integrating audio-visual data for enhanced scene understanding. Despite advancements in state-of-the-art (SOTA) models, we observe three critical flaws: i) The evaluation of the models is mainly focused in sounds produced by objects that are visible in the image, ii) The evaluation often assumes a prior knowledge of the size of the sounding object, and iii) No universal threshold for localization in real-world scenarios is established, as previous approaches only consider positive examples without accounting for both positive and negative cases. In this paper, we introduce a novel test set and metrics designed to complete the current standard evaluation of VSSL models by testing them in scenarios where none of the objects in the image corresponds to the audio input, i.e. a negative audio. We consider three types of negative audio: silence, noise and offscreen. Our analysis reveals that numerous SOTA models fail to appropriately adjust their predictions based on audio input, suggesting that these models may not be leveraging audio information as intended. Additionally, we provide a comprehensive analysis of the range of maximum values in the estimated audio-visual similarity maps, in both positive and negative audio cases, and show that most of the models are not discriminative enough, making them unfit to choose a universal threshold appropriate to perform sound localization without any a priori information of the sounding object, that is, object size and visibility.

A Critical Assessment of Visual Sound Source Localization Models Including Negative Audio

TL;DR

The paper tackles Visual Sound Source Localization (VSSL) in realistic scenarios by introducing negative audio—silence, noise, and offscreen sounds—and showing that existing evaluations overemphasize on-screen sources and prior object information. It introduces extended test sets on the VGG-SS and IS3 datasets and new metrics (pIA, AUC_N, F_LOC, F_AUC) to assess negative-audio performance and global results, plus IoU-based comparisons between localization maps. A universal localization threshold is derived from the distribution of maximum audio-visual similarity, enabling in-the-wild evaluation without prior knowledge of object visibility or size. Empirical results reveal that several SOTA models lack discrimination between positive and negative audio, underscoring the need for more robust VSSL approaches suitable for real-world deployment.

Abstract

The task of Visual Sound Source Localization (VSSL) involves identifying the location of sound sources in visual scenes, integrating audio-visual data for enhanced scene understanding. Despite advancements in state-of-the-art (SOTA) models, we observe three critical flaws: i) The evaluation of the models is mainly focused in sounds produced by objects that are visible in the image, ii) The evaluation often assumes a prior knowledge of the size of the sounding object, and iii) No universal threshold for localization in real-world scenarios is established, as previous approaches only consider positive examples without accounting for both positive and negative cases. In this paper, we introduce a novel test set and metrics designed to complete the current standard evaluation of VSSL models by testing them in scenarios where none of the objects in the image corresponds to the audio input, i.e. a negative audio. We consider three types of negative audio: silence, noise and offscreen. Our analysis reveals that numerous SOTA models fail to appropriately adjust their predictions based on audio input, suggesting that these models may not be leveraging audio information as intended. Additionally, we provide a comprehensive analysis of the range of maximum values in the estimated audio-visual similarity maps, in both positive and negative audio cases, and show that most of the models are not discriminative enough, making them unfit to choose a universal threshold appropriate to perform sound localization without any a priori information of the sounding object, that is, object size and visibility.
Paper Structure (8 sections, 3 figures, 1 table)

This paper contains 8 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. pIA
  4. AUC$_{\text{N}}$
  5. F$_{\text{LOC}}$ and F$_{\text{AUC}}$
  6. $IoU$ between two localization maps
  7. Experiments
  8. Conclusion

Figures (3)

  • Figure 1: Distribution of the max. values of the audio-visual cosine similarities per different models and types of audio (depicted in different colors) across the two different extended test sets: VGGSS (top) and IS3 (bottom). The universal threshold value (Thr) for every model is shown with a solid line.
  • Figure 2: Example of localization results of the different models in both positive and negative audio samples in the extended IS3. The audio-visual similarities below the universal threshold of each model are clipped, then normalized to the interval $[0, 1]$ and overlaid with the original image. The min. and max. values of the audio-visual similarities are reported on top of each image.
  • Figure 3: IoU between different pairs of localization maps: positive and each type of negatives (silence, noise and offscreen sound) and between the negatives themselves. Results of different models depicted in different colors.