Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition

Jay Park, Hong Nguyen, Sean Foley, Jihwan Lee, Yoonjeong Lee, Dani Byrd, Shrikanth Narayanan

TL;DR

The paper addresses phoneme recognition from real-time rtMRI by comparing three representations—raw video, six-channel ROI pixel intensity, and optical flow—for articulatory dynamics. It demonstrates that multimodal fusion (e.g., ROI+Raw Video) yields the best phoneme error rate of $0.34$, while single features underperform relative to their combination. Temporal fidelity analyses show that recognition relies on fine-grained articulatory dynamics, with tongue tip and lip movements contributing most, as revealed by ROI ablation. The work establishes that rtMRI-derived features can provide both accurate and interpretable articulatory representations, offering practical strategies for integrating articulatory data into speech processing pipelines.

Abstract

Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing.

Interpretable Modeling of Articulatory Temporal Dynamics from real-time MRI for Phoneme Recognition

TL;DR

The paper addresses phoneme recognition from real-time rtMRI by comparing three representations—raw video, six-channel ROI pixel intensity, and optical flow—for articulatory dynamics. It demonstrates that multimodal fusion (e.g., ROI+Raw Video) yields the best phoneme error rate of , while single features underperform relative to their combination. Temporal fidelity analyses show that recognition relies on fine-grained articulatory dynamics, with tongue tip and lip movements contributing most, as revealed by ROI ablation. The work establishes that rtMRI-derived features can provide both accurate and interpretable articulatory representations, offering practical strategies for integrating articulatory data into speech processing pipelines.

Abstract

Real-time Magnetic Resonance Imaging (rtMRI) visualizes vocal tract action, offering a comprehensive window into speech articulation. However, its signals are high dimensional and noisy, hindering interpretation. We investigate compact representations of spatiotemporal articulatory dynamics for phoneme recognition from midsagittal vocal tract rtMRI videos. We compare three feature types: (1) raw video, (2) optical flow, and (3) six linguistically-relevant regions of interest (ROIs) for articulator movements. We evaluate models trained independently on each representation, as well as multi-feature combinations. Results show that multi-feature models consistently outperform single-feature baselines, with the lowest phoneme error rate (PER) of 0.34 obtained by combining ROI and raw video. Temporal fidelity experiments demonstrate a reliance on fine-grained articulatory dynamics, while ROI ablation studies reveal strong contributions from tongue and lips. Our findings highlight how rtMRI-derived features provide accuracy and interpretability, and establish strategies for leveraging articulatory data in speech processing.

Paper Structure

This paper contains 15 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methods
  3. Spatial-temporal dynamics via ROI Pixel Intensity
  4. Spatial-temporal dynamics via Optical Flow
  5. Experimental Setup
  6. Dataset
  7. Preprocessing
  8. Training
  9. Evaluation metrics
  10. Results and Discussion
  11. Comparison of features
  12. Temporal fidelity
  13. Ablation: Importance of each ROI
  14. Conclusion and Future Work
  15. Acknowledgement

Figures (3)

  • Figure 1: Overview of three model pipelines for analyzing spatial–temporal articulatory dynamics in rtMRI. (1) Raw video frames, (2) Six-channel ROI pixel intensity, and (3) Optical flow encodes fine-grained articulatory kinematics.
  • Figure 2: Shuffle tests for temporal fidelity evaluation with example of PER and P-Acc calculation.
  • Figure 3: Confusion matrices in percentage of 6-Channel ROI + raw video showing vowels (left) and consonants (right). We ignore the Palatal Consonants /SH/ and /ZH/ since sample size for each is low.