Coding Speech through Vocal Tract Kinematics

TL;DR

This work introduces Speech Articulatory Coding (SPARC), a universal framework that encodes speech into grounded vocal-tract articulatory features and decodes them back into high-quality audio. By leveraging a single-speaker EMA template space and a SSL-based acoustic-to-articulatory inversion, SPARC achieves universal articulatory inference across unseen speakers, while a dedicated speaker-identity encoder disentangles voice texture to enable accent-preserving zero-shot voice conversion. The articulatory synthesizer, based on HiFi-GAN conditioned by a FiLM-modulated speaker embedding, demonstrates high intelligibility and quality across multiple languages and speakers, with strong performance relative to baselines. The framework offers interpretability and controllability of speech production, enabling phonological analyses, articulatory-aware speech synthesis, and potential applications in language learning and therapy, with planned extensions to expressive speech and noisy settings.

Abstract

Vocal tract articulation is a natural, grounded control space of speech production. The spatiotemporal coordination of articulators combined with the vocal source shapes intelligible speech sounds to enable effective spoken communication. Based on this physiological grounding of speech, we propose a new framework of neural encoding-decoding of speech -- Speech Articulatory Coding (SPARC). SPARC comprises an articulatory analysis model that infers articulatory features from speech audio, and an articulatory synthesis model that synthesizes speech audio from articulatory features. The articulatory features are kinematic traces of vocal tract articulators and source features, which are intuitively interpretable and controllable, being the actual physical interface of speech production. An additional speaker identity encoder is jointly trained with the articulatory synthesizer to inform the voice texture of individual speakers. By training on large-scale speech data, we achieve a fully intelligible, high-quality articulatory synthesizer that generalizes to unseen speakers. Furthermore, the speaker embedding is effectively disentangled from articulations, which enables accent-perserving zero-shot voice conversion. To the best of our knowledge, this is the first demonstration of universal, high-performance articulatory inference and synthesis, suggesting the proposed framework as a powerful coding system of speech.

Figures (10)

• Figure 1: The Speech Articulatory Coding (SPARC) framework. It encodes speech using articulatory features (analysis) and decodes these features back into speech (synthesis).
• Figure 2: Pipeline of articulatory analysis and synthesis. The articulatory analysis is composed of vocal tract articulation, source features, and speaker embedding, which are then fed to the synthesizer (HiFi-GAN) in the synthesis pipeline. The modules colored with orange (FFN and HiFi-GAN) are updated while training the synthesis model and other modules are fixed.
• Figure 3: Examples of SSL-linear prediction on MNGU0 (left), transformed prediction from MNGU0 to a female MOCHA speaker (middle) and to a male HPRC speaker (right). Predictions are denoted with the colored lines and ground truths are denoted with the black dotted lines.
• Figure 4: Visualization by T-SNE of utterance-wise averaged articulations (left) and speaker embeddings (right) from 6 different VCTK speakers. The perplexity is set as 10.
• Figure 5: Articulatory traces encoded for speaking "lock" and "rock", denoted by the different line styles, "--" and "--", respectively. The bottom panel shows the midsagittal displacements of TT, TB, and TD, and the top panels show snapshots of the corresponding vocal tract anatomy. In the snapshots, the vocal tract of "lock" and "rock" are overlaid with separate colors, orange and pink, respectively. The shaded region indicates the window of "l" or "r". The color is darkened while interpolating from "lock" to "rock", where the line style indicates the recognized words.