Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI

Neil Shah, Ayan Kashyap, Shirish Karande, Vineet Gandhi

TL;DR

This work tackles rtMRI-based speech synthesis by decoupling content from noisy audio targets. It introduces MRI2Speech, which first predicts textual content from silent rtMRI using a fine-tuned, multi-modal AV-HuBERT, and then aligns articulatory timing with a novel flow-based duration predictor to drive a speech decoder. The method enables synthesis in novel voices by transferring learned speaker alignment to a LJSpeech-based VITS, achieving a WER of 15.18% on USC-TIMIT and demonstrating strong unseen-speaker generalization across USC-TIMIT and ASD1. The approach advances robust articulator-to-speech synthesis, with implications for silent-speech interfaces and assistive communication across diverse speakers and hardware settings.

Abstract

Previous real-time MRI (rtMRI)-based speech synthesis models depend heavily on noisy ground-truth speech. Applying loss directly over ground truth mel-spectrograms entangles speech content with MRI noise, resulting in poor intelligibility. We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI and incorporates a new flow-based duration predictor for speaker-specific alignment. The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice. We conduct thorough experiments on two datasets and demonstrate our method's generalization ability to unseen speakers. We assess our framework's performance by masking parts of the rtMRI video to evaluate the impact of different articulators on text prediction. Our method achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art. Speech samples are available at https://mri2speech.github.io/MRI2Speech/

MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI

TL;DR

This work tackles rtMRI-based speech synthesis by decoupling content from noisy audio targets. It introduces MRI2Speech, which first predicts textual content from silent rtMRI using a fine-tuned, multi-modal AV-HuBERT, and then aligns articulatory timing with a novel flow-based duration predictor to drive a speech decoder. The method enables synthesis in novel voices by transferring learned speaker alignment to a LJSpeech-based VITS, achieving a WER of 15.18% on USC-TIMIT and demonstrating strong unseen-speaker generalization across USC-TIMIT and ASD1. The approach advances robust articulator-to-speech synthesis, with implications for silent-speech interfaces and assistive communication across diverse speakers and hardware settings.

Abstract

Previous real-time MRI (rtMRI)-based speech synthesis models depend heavily on noisy ground-truth speech. Applying loss directly over ground truth mel-spectrograms entangles speech content with MRI noise, resulting in poor intelligibility. We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI and incorporates a new flow-based duration predictor for speaker-specific alignment. The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice. We conduct thorough experiments on two datasets and demonstrate our method's generalization ability to unseen speakers. We assess our framework's performance by masking parts of the rtMRI video to evaluate the impact of different articulators on text prediction. Our method achieves a Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art. Speech samples are available at https://mri2speech.github.io/MRI2Speech/

Paper Structure

This paper contains 10 sections, 1 equation, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Fine-tuning AV-HuBERT for text prediction
  4. Training VITS for duration estimation
  5. Speech Synthesis
  6. Datasets
  7. Results and Discussion
  8. Recognition performance on predicted text
  9. Recognition performance on synthesized speech
  10. Conclusions

Figures (2)

  • Figure 1: System diagram of the inference process: AV-HuBERT predicts text from silent rtMRI videos. A trained duration predictor expands the phoneme representations, which are mapped to the acoustic space using a normalizing flow, allowing the decoder to synthesize speech.
  • Figure 2: Mel-spectrograms of (A) original speech, and synthesized speech using (B) Otani et al.'s otani23_interspeech approach and (C) our proposed method. The region marked by the white dashed box highlights the inability of current methods to accurately estimate essential formants.