Normalization through Fine-tuning: Understanding Wav2vec 2.0 Embeddings for Phonetic Analysis
Yiming Wang, Yi Yang, Jiahong Yuan
TL;DR
This work investigates how phonetic normalization emerges in transformer-based speech models, focusing on wav2vec 2.0. By fine-tuning wav2vec 2.0 for framewise tone, phone, and sex classification on Mandarin AISHELL-1 and analyzing layer-wise embeddings with SVCCA and UMAP, the study demonstrates that normalization arises as task-irrelevant information is selectively suppressed while task-relevant features are enhanced. It further shows that multitask fine-tuning can encode multiple attributes (e.g., tone and sex) without sacrificing performance, challenging the notion that normalization requires suppressing all non-target information. These findings offer practical guidance for phonetic analysis, suggesting flexible normalization mechanisms in deep speech models and implications for understanding human speech perception.
Abstract
Phonetic normalization plays a crucial role in speech recognition and analysis, ensuring the comparability of features derived from raw audio data. However, in the current paradigm of fine-tuning pre-trained large transformer models, phonetic normalization is not deemed a necessary step; instead, it is implicitly executed within the models. This study investigates the normalization process within transformer models, especially wav2vec 2.0. Through a comprehensive analysis of embeddings from models fine-tuned for various tasks, our results demonstrate that fine-tuning wav2vec 2.0 effectively achieves phonetic normalization by selectively suppressing task-irrelevant information. We found that models fine-tuned for multiple tasks retain information for both tasks without compromising performance, and that suppressing task-irrelevant information is not necessary for effective classification. These findings provide new insights into how phonetic normalization can be flexibly achieved in speech models and how it is realized in human speech perception.