A Speech-to-Video Synthesis Approach Using Spatio-Temporal Diffusion for Vocal Tract MRI

TL;DR

This work tackles the problem of generating realistic vocal-tract MRI visuals directly from speech. It introduces Speech2-MRI, a spatio-temporal latent diffusion framework that conditions MRI video synthesis on audio, leveraging a VAE-based video latent space and Wav2Vec-derived speech embeddings. Across USC 75-Speaker RT-MRI data and tongue-cancer cine-MRI data, the STDiff variant (especially with PNDM scheduling) achieves strong objective metrics and good generalization to unseen speakers and words, with human evaluators noting good audio-visual alignment and transitions. The approach holds promise for outpatient therapy and personalized vocal-tract simulations, while future work aims to reduce data demands and incorporate phonetic constraints to further improve articulatory fidelity.

Abstract

Understanding the relationship between vocal tract motion during speech and the resulting acoustic signal is crucial for aided clinical assessment and developing personalized treatment and rehabilitation strategies. Toward this goal, we introduce an audio-to-video generation framework for creating Real Time/cine-Magnetic Resonance Imaging (RT-/cine-MRI) visuals of the vocal tract from speech signals. Our framework first preprocesses RT-/cine-MRI sequences and speech samples to achieve temporal alignment, ensuring synchronization between visual and audio data. We then employ a modified stable diffusion model, integrating structural and temporal blocks, to effectively capture movement characteristics and temporal dynamics in the synchronized data. This process enables the generation of MRI sequences from new speech inputs, improving the conversion of audio into visual data. We evaluated our framework on healthy controls and tongue cancer patients by analyzing and comparing the vocal tract movements in synthesized videos. Our framework demonstrated adaptability to new speech inputs and effective generalization. In addition, positive human evaluations confirmed its effectiveness, with realistic and accurate visualizations, suggesting its potential for outpatient therapy and personalized simulation of vocal tract visualizations.

Figures (5)

• Figure 1: Methodology proposed in this study, where a diffusion model designed for generating MRI sequences, detailing its workflow in three main stages. Initially, MRI frames and corresponding audio waveforms are input for training, while for inference only the audio is considered. In the diffusion phase, these inputs are transformed into a latent space and progressively noised before being incrementally denoised. The process culminates in the output of an MRI sequence.
• Figure 2: Temporal alignment of speech samples and image frames, where it is illustrated how overlapping windows of images and speech segments are synchronized for effective analysis. Each image frame corresponds to approximately 67 ms of speech. To capture temporal context, bidirectional segments spanning three consecutive windows (approximately 200 ms) are used.
• Figure 3: General structure of the spatio--temporal latent diffusion architecture used to synthesize speech to MRI sequences. It consists of three main steps. (1) Video encoding by using a VAE architecture, which converts the video into a latent matrix representation. (2) The diffusion process, where seeded noise is added to the latent matrix (latent seed) which is denoised by a spatio-temporal UNet conditioned by the latent representation obtained from the audio encoder. (3) The decoding process, which used the decoder of the pretrained VAE to generate the final output. Notice that the video encoding is removed during inference.
• Figure 4: Comparison of the models used in this work for a female participant when saying "When the". The videos were resample to 5 fps only for demonstration purposes.
• Figure 5: Comparison of the models employed in this work for a patient pronouncing the word "a souk". The videos were resample to 5 fps only for demonstration purposes.