Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Acoustic-to-articulatory inversion for dysarthric speech: Are pre-trained self-supervised representations favorable?

Sarthak Kumar Maharana, Krishna Kamal Adidam, Shoumik Nandi, Ajitesh Srivastava

TL;DR

This work tackles acoustic-to-articulatory inversion for dysarthric speech by leveraging pre-trained self-supervised representations, conditioned on speaker embeddings, to improve articulatory trajectory prediction under low-resource conditions. Using a BLSTM regressor and the TORGO dataset, the authors compare multiple SSL features (including wav2vec, APC, NPC, DeCoAR, TERA, vq_wav2vec, Mockingjay) against MFCC baselines across seen and unseen subjects and three training schemes. Key findings show DeCoAR (especially with fine-tuning) and wav2vec consistently outperform MFCCs, with notable gains in unseen subjects and with minimal target-subject data; SSL features also deliver strong articulator-specific improvements. The results suggest SSL-based representations provide robust, generalizable acoustic features for dysarthric AAI, with potential to support clinical assessment and therapy planning, and point to future work on language-mismatched and higher-severity dysarthric data.

Abstract

Acoustic-to-articulatory inversion (AAI) involves mapping from the acoustic to the articulatory space. Signal-processing features like the MFCCs, have been widely used for the AAI task. For subjects with dysarthric speech, AAI is challenging because of an imprecise and indistinct pronunciation. In this work, we perform AAI for dysarthric speech using representations from pre-trained self-supervised learning (SSL) models. We demonstrate the impact of different pre-trained features on this challenging AAI task, at low-resource conditions. In addition, we also condition x-vectors to the extracted SSL features to train a BLSTM network. In the seen case, we experiment with three AAI training schemes (subject-specific, pooled, and fine-tuned). The results, consistent across training schemes, reveal that DeCoAR, in the fine-tuned scheme, achieves a relative improvement of the Pearson Correlation Coefficient (CC) by ~1.81% and ~4.56% for healthy controls and patients, respectively, over MFCCs. We observe similar average trends for different SSL features in the unseen case. Overall, SSL networks like wav2vec, APC, and DeCoAR, trained with feature reconstruction or future timestep prediction tasks, perform well in predicting dysarthric articulatory trajectories.

Acoustic-to-articulatory inversion for dysarthric speech: Are pre-trained self-supervised representations favorable?

TL;DR

This work tackles acoustic-to-articulatory inversion for dysarthric speech by leveraging pre-trained self-supervised representations, conditioned on speaker embeddings, to improve articulatory trajectory prediction under low-resource conditions. Using a BLSTM regressor and the TORGO dataset, the authors compare multiple SSL features (including wav2vec, APC, NPC, DeCoAR, TERA, vq_wav2vec, Mockingjay) against MFCC baselines across seen and unseen subjects and three training schemes. Key findings show DeCoAR (especially with fine-tuning) and wav2vec consistently outperform MFCCs, with notable gains in unseen subjects and with minimal target-subject data; SSL features also deliver strong articulator-specific improvements. The results suggest SSL-based representations provide robust, generalizable acoustic features for dysarthric AAI, with potential to support clinical assessment and therapy planning, and point to future work on language-mismatched and higher-severity dysarthric data.

Abstract

Acoustic-to-articulatory inversion (AAI) involves mapping from the acoustic to the articulatory space. Signal-processing features like the MFCCs, have been widely used for the AAI task. For subjects with dysarthric speech, AAI is challenging because of an imprecise and indistinct pronunciation. In this work, we perform AAI for dysarthric speech using representations from pre-trained self-supervised learning (SSL) models. We demonstrate the impact of different pre-trained features on this challenging AAI task, at low-resource conditions. In addition, we also condition x-vectors to the extracted SSL features to train a BLSTM network. In the seen case, we experiment with three AAI training schemes (subject-specific, pooled, and fine-tuned). The results, consistent across training schemes, reveal that DeCoAR, in the fine-tuned scheme, achieves a relative improvement of the Pearson Correlation Coefficient (CC) by ~1.81% and ~4.56% for healthy controls and patients, respectively, over MFCCs. We observe similar average trends for different SSL features in the unseen case. Overall, SSL networks like wav2vec, APC, and DeCoAR, trained with feature reconstruction or future timestep prediction tasks, perform well in predicting dysarthric articulatory trajectories.
Paper Structure (9 sections, 4 figures, 4 tables)

This paper contains 9 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Dataset
  3. Proposed Methodology
  4. Experimental Setup
  5. Results and Discussions
  6. Effect of speaker-specific embeddings
  7. Impact of different pre-trained SSL features
  8. Articulatory specific results
  9. Conclusions

Figures (4)

  • Figure 1: Block diagram of the proposed AAI model with pre-trained SSL features as the input acoustic features, conditioned with x-vectors.
  • Figure 2: t-SNE plot of x-vector speaker embeddings of healthy controls (MC01, MC04) and patients (F03, F04), used in this work.
  • Figure 3: Average CC (with error bars) for unseen target subjects using MFCCs and DeCoAR at varying amounts of training data (t%), averaged across all the articulators, fold, and sentences. t = 0 refers to the unseen subject evaluation.
  • Figure 4: CC values for each articulator predicted using MFCCs and DeCoAR features for healthy controls and patients, in the fine-tuned training scheme.