Audio Outperforms Text for Visual Decoding
Zhengdi Zhang, Hao Zhang, Wenjun Xia
TL;DR
This work challenges the conventional reliance on text for brain–language semantic decoding by showing that auditory semantic representations, derived from CLAP, yield higher zero-shot visual decoding accuracy and greater efficiency than text when aligned with EEG and image data. By building a brain–vision–audio VAE with MoPoE fusion and mutual information regularization, the approach demonstrates stronger neural alignment and generalization across subjects. The results support cognitive theories that auditory modalities capture richer, temporally structured semantics that better reflect neural processing, with practical implications for more natural and efficient brain–computer interfaces. Overall, replacing text with auditory semantics advances both scientific understanding of brain-language coupling and the development of cognitively plausible BCI systems.
Abstract
Decoding visual semantic representations from human brain activity is a significant challenge. While recent zero-shot decoding approaches have improved performance by leveraging aligned image-text datasets, they overlook a fundamental aspect of human cognition: semantic understanding is inherently anchored in the auditory modality of speech, not text. To address this, our study introduces the first comparative framework for evaluating auditory versus textual semantic modalities in zero-shot visual neural decoding. We propose a novel brain-visual-auditory multimodal alignment model that directly utilizes auditory representations to encapsulate semantics, serving as a substitute for traditional textual descriptors. Our experimental results demonstrate that the auditory modality not only surpasses the textual modality in decoding accuracy but also achieves higher computational efficiency. These findings indicate that auditory semantic representations are more closely aligned with neural activity patterns during visual processing. This work reveals the critical and previously underestimated role of auditory semantics in decoding visual cognition and provides new insights for developing brain-computer interfaces that are more congruent with natural human cognitive mechanisms.