Towards Dynamic Neural Communication and Speech Neuroprosthesis Based on Viseme Decoding

TL;DR

A diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication and bridge the gap between brain signals and dynamic visual interfaces is developed.

Abstract

Decoding text, speech, or images from human neural signals holds promising potential both as neuroprosthesis for patients and as innovative communication tools for general users. Although neural signals contain various information on speech intentions, movements, and phonetic details, generating informative outputs from them remains challenging, with mostly focusing on decoding short intentions or producing fragmented outputs. In this study, we developed a diffusion model-based framework to decode visual speech intentions from speech-related non-invasive brain signals, to facilitate face-to-face neural communication. We designed an experiment to consolidate various phonemes to train visemes of each phoneme, aiming to learn the representation of corresponding lip formations from neural signals. By decoding visemes from both isolated trials and continuous sentences, we successfully reconstructed coherent lip movements, effectively bridging the gap between brain signals and dynamic visual interfaces. The results highlight the potential of viseme decoding and talking face reconstruction from human neural signals, marking a significant step toward dynamic neural communication systems and speech neuroprosthesis for patients.

Figures (5)

• Figure 1: Overview of the proposed viseme decoding framework from EEG signals. The recorded EEG signals are preprocessed in two ways for training. Firstly, sentence-level EEG signals are segmented at the phoneme-level based on real audio of uttered sentences to extract phoneme-based viseme feature embeddings. Secondly, single-trial EEG data of viseme classes are preprocessed and given as input features. DDPM in the proposed framework learns relevant features by injecting noise and restoring the data, while the encoder and decoder work together to compensate for the information loss during this process, allowing the accurate reconstruction of EEG signals. The trained model then predicts the viseme classes by passing the compressed information from the final encoder stage through a classifier. Finally, the predicted viseme classes are sequentially reconstructed into articulatory movements of sentences.
• Figure 2: Experimental paradigm for EEG signals recording. The phonemes and sentences were presented randomly to guide overt, mimed, and imagined speech. Participants were instructed to perform the experiment with a focus on the changes in lip shapes and articulator movements.
• Figure 3: The projection of phoneme encodings into a 2D space using t-SNE shows that phoneme classes representing visemes clustered together, with class groups exhibiting similar lip shapes positioned relatively close to each other.
• Figure 4: The confusion matrices of the classification results across overt, mimed, and imagined speech. The results indicated that overt speech displays more distinguishable patterns. Even in mimed and imagined speech, where there was no audible speech and reduced articulatory cues, our approach could capture and differentiate subtle patterns, supporting the proposed framework.
• Figure 5: The recorded sentence data were epoched into phoneme-viseme segments based on the audio, which were used to fine-tune and adapt to the single trial-based trained model. The fine-tuned model, sharing similar articulatory and lip movement features, provided improved viseme decoding performance from continuous sentence-level brain signals. This allowed for more accurate reaching of the target sentence.