TIPAA-SSL: Text Independent Phone-to-Audio Alignment based on Self-Supervised Learning and Knowledge Transfer
Noé Tits, Prernna Bhatnagar, Thierry Dutoit
TL;DR
The paper addresses text-independent phone-to-audio alignment with a language-independent approach suitable for pronunciation feedback in language learning. It combines wav2vec2-based self-supervised phoneme representations, PCA dimensionality reduction, and a frame-level classifier to generate framewise phoneme probabilities and boundaries, aiming for cross-dialect generalization. Empirical results on TIMIT and SCRIBE show competitive or superior performance to the state-of-the-art CharSiU method, with strong generalization to British English and potential applicability to other languages. This approach advances multilingual phonetic alignment with practical implications for CALL and speech processing systems, while highlighting the need for broader non-native data in future work.
Abstract
In this paper, we present a novel approach for text independent phone-to-audio alignment based on phoneme recognition, representation learning and knowledge transfer. Our method leverages a self-supervised model (wav2vec2) fine-tuned for phoneme recognition using a Connectionist Temporal Classification (CTC) loss, a dimension reduction model and a frame-level phoneme classifier trained thanks to forced-alignment labels (using Montreal Forced Aligner) to produce multi-lingual phonetic representations, thus requiring minimal additional training. We evaluate our model using synthetic native data from the TIMIT dataset and the SCRIBE dataset for American and British English, respectively. Our proposed model outperforms the state-of-the-art (charsiu) in statistical metrics and has applications in language learning and speech processing systems. We leave experiments on other languages for future work but the design of the system makes it easily adaptable to other languages.