Speaker- and Text-Independent Estimation of Articulatory Movements and Phoneme Alignments from Speech

TL;DR

The paper tackles the problem of jointly estimating articulatory movements and phoneme alignments from speech in a speaker- and text-independent setting by introducing APTAI, which blends acoustic-to-articulatory inversion with phoneme-to-articulatory motion estimation. It evaluates two multi-task learning strategies: a frame-classification based approach (APTAI) and a two-stage forced-alignment approach (f-APTAI), both built on wav2vec2 representations. On the HPRC dataset with LOSO testing, APTAI achieves a mean TV correlation of $PCC=0.73$ and frame overlap of $87.38\%$, with frame-classification outperforming forced alignment in alignment quality. The work demonstrates that end-to-end APTAI is feasible with competitive articulatory and phoneme metrics, while highlighting areas for improvement in phoneme alignment and suggesting future enhancements such as higher frame-rate processing to reach $100\,\text{Hz}$ TV regression. The methods hold promise for applications in speech therapy and articulatory research that require robust, speaker- and text-independent analysis.

Abstract

This paper introduces a novel combination of two tasks, previously treated separately: acoustic-to-articulatory speech inversion (AAI) and phoneme-to-articulatory (PTA) motion estimation. We refer to this joint task as acoustic phoneme-to-articulatory speech inversion (APTAI) and explore two different approaches, both working speaker- and text-independently during inference. We use a multi-task learning setup, with the end-to-end goal of taking raw speech as input and estimating the corresponding articulatory movements, phoneme sequence, and phoneme alignment. While both proposed approaches share these same requirements, they differ in their way of achieving phoneme-related predictions: one is based on frame classification, the other on a two-staged training procedure and forced alignment. We reach competitive performance of 0.73 mean correlation for the AAI task and achieve up to approximately 87% frame overlap compared to a state-of-the-art text-dependent phoneme force aligner.

Figures (6)

• Figure 1: Nine tract variables (TVs), used for speaker-independent articulatory speech inversion. Adapted from chartier2018encodingwu2023speakerindependent.
• Figure 2: Proposed APTAI model, based on wav2vec2.0 fine-tuning via frame-classification and TV regression.
• Figure 3: Proposed f-APTAI model, based on TV regression and a two-staged forced alignment via cross-attention.
• Figure 4: PCA of the embeddings from the best-performing fine-tuned phoneme recognition model from Table \ref{['tab:phoneme_recognizer_fine_tuning_results']}.
• Figure 5: Example model prediction ( APTAI) and ground truth for an unseen speaker (only a selection of all TVs is shown).