Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization

TL;DR

The paper tackles how cross-modal conflicts affect sound localization in AI versus humans and introduces AudioCOCO, a large, depth-aware stereo audio–visual dataset, along with EchoPin, a neuroscience-inspired SSL model. Through six testing conditions and human psychophysics, it shows humans maintain robust localization under conflicting or missing visuals, while AI models often default to visual cues. EchoPin, trained on AudioCOCO with HRTF-based processing and cochlear-inspired representations, outperforms prior models and exhibits human-like horizontal bias, highlighting the importance of high-fidelity sensory priors. The findings emphasize the need for physics-grounded data and stereo spatial cues to build more robust multimodal systems with real-world relevance.

Abstract

Imagine hearing a dog bark and turning toward the sound only to see a parked car, while the real, silent dog sits elsewhere. Such sensory conflicts test perception, yet humans reliably resolve them by prioritizing sound over misleading visuals. Despite advances in multimodal AI integrating vision and audio, little is known about how these systems handle cross-modal conflicts or whether they favor one modality. In this study, we systematically examine modality bias and conflict resolution in AI sound localization. We assess leading multimodal models and benchmark them against human performance in psychophysics experiments across six audiovisual conditions, including congruent, conflicting, and absent cues. Humans consistently outperform AI, demonstrating superior resilience to conflicting or missing visuals by relying on auditory information. In contrast, AI models often default to visual input, degrading performance to near chance levels. To address this, we propose a neuroscience-inspired model, EchoPin, which uses a stereo audio-image dataset generated via 3D simulations. Even with limited training data, EchoPin surpasses existing benchmarks. Notably, it also mirrors human-like horizontal localization bias favoring left-right precision-likely due to the stereo audio structure reflecting human ear placement. These findings underscore how sensory input quality and system architecture shape multimodal representation accuracy.

Figures (18)

• Figure 1: (a) Sound source localization challenge in naturalistic images. In the multi-source scenario (orange panel), multiple objects in the scene—such as pedestrians, birds, cars, and trucks (highlighted by yellow boxes)—emit sounds, whereas in the single-source scenario (red panel), only one object, a telephone (red box), produces sound. In both settings, the task is to localize all sounding objects based on two-channel stereo audio. To allow systematic and controllable benchmarking of human and AI performance, we focus on the single-source localization task (red panel), which remains challenging due to scene clutter, occlusions, and ambiguous visual cues. (b) Depth-aware stereo audio synthesis. In the 3D simulator, a human listener (interaural distance: 0.17m) is placed at the origin, facing the RGB image on the screen. This image and its depth image are aligned in the same direction. Using a spatial audio renderer and audio from our library, stereo audio of the target car (red box) in the RGB image can be synthesized (see Sec \ref{['subsec:audiococo']}).
• Figure 2: Overview of congruent and manipulated vision-audio conditions in our UniAV framework and task schematic. An example of the vision-audio congruent condition (a) is shown, where a dog sound is played (black box) with a matching visual source. Visual and auditory modifications for the other five experimental conditions (b-f) are also displayed. See Sec \ref{['subsec:condition']} for more details. (h) Each trial began with a fixation cross (500 ms), followed by the presentation of an image-audio pair from either of the six conditions (a-f). Participants were instructed to use the computer mouse to click on the perceived location of the sound source within 20 seconds.
• Figure 3: Overview of our neuroscience-inspired EchoPin model. EchoPin takes as input a static image paired with a two-channel stereo audio signal. The stereo waveforms are first filtered using the Head-Related Transfer Function (HRTF) to simulate sound filtering by the pinnae, and then converted into cochleagrams to mimic auditory processing in the human cochlea. The audio and visual streams are independently processed through dedicated encoders. During training, semantic alignment between the two modalities is enforced using a contrastive loss applied to paired audio and visual embeddings, while localization alignment is achieved by regressing the predicted sound source location (indicated by a red triangle) from the multimodal feature similarity map to the ground-truth location (red bounding box).
• Figure 4: Object size matters for humans and AI models in the congruent condition. Accuracy increases with object sizes for both humans and AI models, with humans outperforming AI models, especially for small targets. Here and in subsequent figures, error bars represent Standard Error Mean (SEM).
• Figure 5: ConflictVCue and AbsVCue hurt SSL performance for both humans and AI models. The accuracy of humans and AI models under ConflicVcue and AbsVcue conditions is shown for object size 2, with results from the congruent condition included for comparison. See Supp Tab. \ref{['tab:all']} for the results on object sizes 1 and 3.