Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Segmentation for Vocal Tract Modeling

Rishi Jain, Bohan Yu, Peter Wu, Tejas Prabhune, Gopala Anumanchipalli

TL;DR

Problem: occlusion of internal articulators and limited labeled RT-MRI data hinder vocal tract modeling. Approach: develop a vision-based Heatmap U-Net and a multimodal Transformer that integrate audio to segment RT-MRI frames, and release labels for a 75-speaker dataset. Contributions: KL-divergence loss with articulatory weighting improves segmentation; the Speech MRI Open Dataset is labeled to provide over 20 hours of data (9x increase); multimodal fusion improves segmentation and downstream speech synthesis, outperforming the baseline. Significance: enables scalable, interpretable articulatory representations for speech processing and linguistics research.

Abstract

Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}.

Multimodal Segmentation for Vocal Tract Modeling

TL;DR

Problem: occlusion of internal articulators and limited labeled RT-MRI data hinder vocal tract modeling. Approach: develop a vision-based Heatmap U-Net and a multimodal Transformer that integrate audio to segment RT-MRI frames, and release labels for a 75-speaker dataset. Contributions: KL-divergence loss with articulatory weighting improves segmentation; the Speech MRI Open Dataset is labeled to provide over 20 hours of data (9x increase); multimodal fusion improves segmentation and downstream speech synthesis, outperforming the baseline. Significance: enables scalable, interpretable articulatory representations for speech processing and linguistics research.

Abstract

Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}.
Paper Structure (14 sections, 4 figures, 3 tables)

This paper contains 14 sections, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Datasets
  3. USC-TIMIT Dataset
  4. Speech MRI Open Dataset
  5. Models and Training
  6. Frequency-domain Gradient Descent Baseline
  7. Heatmap U-Net
  8. Multimodal Transformer
  9. Results
  10. Vision-only U-Net
  11. Labeling the Speech MRI Open Dataset
  12. Comparison with Multimodal Transformer
  13. Qualitative Evaluation
  14. Conclusion

Figures (4)

  • Figure 1: The attention U-Net model. Dotted lines represent the paths of attention gating in contracting/expanding layers.
  • Figure 2: Architecture of the multimodal segmentation model.
  • Figure 3: Two representative examples of predicted MRI points (right) compared to expert hand labels (left). The examples are spoken by unseen Female (bottom) and Male (top) speakers in the Speech MRI Open Dataset.
  • Figure 4: L1 losses [↓] (left) and Pearson Correlation Coefficients (PCCs) [↑] (right) comparing MRI trajectories of unseen examples from seen speakers of a given model with the USC-TIMIT ground truth. Varying through a subset of six representative models.