Artificial Neural Networks Trained on Noisy Speech Exhibit the McGurk Effect

TL;DR

This work investigates how auditory and visual speech integration, exemplified by the McGurk effect, can emerge in artificial neural networks under development-like learning conditions. By evaluating several AV speech models and a self-supervised CPC-based network, the authors show that noise during training amplifies audiovisual fusion, and that McGurk-like percepts can arise even when networks are trained only on congruent speech. A systematic exploration across noise levels reveals a finite range in which training noise promotes integration, with extreme noise causing breakdown of audiovisual fusion. The findings support using ANNs as proxy models for perception and development, while highlighting differences from human behavior and suggesting that deeper architectures may be required to fully capture human-like multisensory speech processing.

Abstract

Humans are able to fuse information from both auditory and visual modalities to help with understanding speech. This is demonstrated through a phenomenon known as the McGurk Effect, during which a listener is presented with incongruent auditory and visual speech that fuse together into the percept of illusory intermediate phonemes. Building on a recent framework that proposes how to address developmental 'why' questions using artificial neural networks, we evaluated a set of recent artificial neural networks trained on audiovisual speech by testing them with audiovisually incongruent words designed to elicit the McGurk effect. We show that networks trained entirely on congruent audiovisual speech nevertheless exhibit the McGurk percept. We further investigated 'why' by comparing networks trained on clean speech to those trained on noisy speech, and discovered that training with noisy speech led to a pronounced increase in both visual responses and McGurk responses across all models. Furthermore, we observed that systematically increasing the level of auditory noise during ANN training also increased the amount of audiovisual integration up to a point, but at extreme noise levels, this integration failed to develop. These results suggest that excessive noise exposure during critical periods of audiovisual learning may negatively influence the development of audiovisual speech integration. This work also demonstrates that the McGurk effect reliably emerges untrained from the behaviour of both supervised and unsupervised networks, even networks trained only on congruent speech. This supports the notion that artificial neural networks might be useful models for certain aspects of perception and cognition.

Figures (4)

• Figure 1: McGurk Stimuli Classification Results on Audiovisual Input. Networks were provided both auditory and visual input, and human participants listened to speech and were presented with a video of the speaker on a computer screen. Networks showed increased McGurk and visual responses when trained with auditory noise relative to their counterparts trained on clean speech.
• Figure 2: Audio Only Control Stimuli Classification Results. Networks were provided with auditory input and zeros for visual input, and human participants listened to speech and were presented with a blank screen. Networks primarily responded with auditory option and humans selected the auditory and McGurk options 25.1% and 74.4% of the time respectively.
• Figure 3: AudioVisual CPC Network Performance Under Increased Levels of Training Noise. Networks were trained with systematically increasing levels of babble noise in audio and were tested using clean audio and video. McGurk and visual responses increased with increased training noise, up to -5 dB SNR. At -10 dB and higher audiovisual integration broke and the network always selected the auditory response. Error bars indicate 95% confidence intervals calculated using bootstrapping across the 10 cross-validation folds of the k-NN classifier.
• Figure 4: AudioVisual CPC Embedding Distance Stripplot under Increased Levels of Training Noise. Distances are the average cosine-DTW distance from each incongruent McGurk stimuli embedding to all word embeddings corresponding to the auditory option, visual option, and audiovisual option within each word pairing. The % McGurk response from Figure \ref{['figure:noise']} is replotted for reference.