Improving Spoken Language Modeling with Phoneme Classification: A Simple Fine-tuning Approach

TL;DR

It is shown that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

Abstract

Recent progress in Spoken Language Modeling has shown that learning language directly from speech is feasible. Generating speech through a pipeline that operates at the text level typically loses nuances, intonations, and non-verbal vocalizations. Modeling directly from speech opens up the path to more natural and expressive systems. On the other hand, speech-only systems require up to three orders of magnitude more data to catch up to their text-based counterparts in terms of their semantic abilities. We show that fine-tuning speech representation models on phoneme classification leads to more context-invariant representations, and language models trained on these units achieve comparable lexical comprehension to ones trained on hundred times more data.

Figures (4)

• Figure 1: Trade-off between language modeling and expressive resynthesis. *: embeddings initialized from unit centroids.
• Figure 2: ABX error rate averaged across subset (dev-clean, dev-other) and speaker (within, across) conditions.
• Figure 3: ABX error rate for models finetuned with CTC, averaged across subset and speaker conditions.
• Figure 4: Difference between the MCD of the fine-tuned models and Base L11 on Expresso for each style.