An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data

TL;DR

This work tackles acoustic-to-articulatory inversion (AAI) for ultrasound tongue imaging (UTI) by introducing a diffusion-based framework that jointly leverages individual acoustic cues and universal textual information. It presents two pipelines: an audio-only diffusion path and an audio-textual diffusion path that fuses wav2vec 2.0 embeddings with BERT-derived text via cross-attention to condition the diffusion process, including a cascade denoising scheme. Evaluated on a Mandarin speech-ultrasound dataset, the audio-textual model markedly improves perceptual and distributional quality (e.g., LPIPS and FID) over a state-of-the-art DNN baseline, delivering UTI data with clearer tongue contours. The approach reduces reliance on large parallel datasets and offers a data-efficient tool for linguistic analysis and clinical tongue-function assessment.

Abstract

Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only using the personalized acoustic information to derive the general patterns of tongue motions, and thus the quality of generated UTI data is limited. To address this issue, this paper proposes an audio-textual diffusion model for the UTI data generation task. In this model, the inherent acoustic characteristics of individuals related to the tongue motion details are encoded by using wav2vec 2.0, while the ASR transcriptions related to the universality of tongue motions are encoded by using BERT. UTI data are then generated by using a diffusion module. Experimental results showed that the proposed diffusion model could generate high-quality UTI data with clear tongue contour that is crucial for the linguistic analysis and clinical assessment. The project can be found on the website\footnote{https://yangyudong2020.github.io/wav2uti/

Figures (2)

• Figure 1: The overall structure of the audio-only and audio-textual diffusion models for ultrasound tongue imaging (UTI) data generation, where $f_c$ is the fusion conditions from the acoustic and textual encoding.
• Figure 2: Examples of real and generated UTI data.