Personalized Neural Speech Codec

TL;DR

The paper tackles the challenge of high-complexity neural speech codecs by introducing personalization to neural speech coding (PNSC). It learns speaker embeddings with a Siamese network on LibriSpeech, clusters speakers into C groups, and trains per-group LPCNet decoders (PLPCNet) to specialize the decoding process for each speaker group. The approach demonstrates that PLPCNet achieves perceptual quality comparable to or better than a larger generic LPCNet at the same bitrate, while reducing decoder parameters by up to roughly 36% for smaller models. This work highlights the value of diversity-aware, group-specific decoders for efficient neural speech coding and motivates extending personalization to other NSC models and bitrate regimes.

Abstract

In this paper, we propose a personalized neural speech codec, envisioning that personalization can reduce the model complexity or improve perceptual speech quality. Despite the common usage of speech codecs where only a single talker is involved on each side of the communication, personalizing a codec for the specific user has rarely been explored in the literature. First, we assume speakers can be grouped into smaller subsets based on their perceptual similarity. Then, we also postulate that a group-specific codec can focus on the group's speech characteristics to improve its perceptual quality and computational efficiency. To this end, we first develop a Siamese network that learns the speaker embeddings from the LibriSpeech dataset, which are then grouped into underlying speaker clusters. Finally, we retrain the LPCNet-based speech codec baselines on each of the speaker clusters. Subjective listening tests show that the proposed personalization scheme introduces model compression while maintaining speech quality. In other words, with the same model complexity, personalized codecs produce better speech quality.

Figures (5)

• Figure 1: The overview of the personalized NSC system used in the speech communication scenario. Note that the utterance encoder model $\mathcal{G}(\cdot)$ could be also specialized to handle a specific speaker group, while we leave that option to future work.
• Figure 2: A simplified LPCNet architecture. PLPCNet controls its complexity by adjusting the number of the GRU layer's hidden units.
• Figure 3: Clustering of speakers from different choices of $C$.
• Figure 4: Results of the listening test. 95% confidence intervals are shown as upper and lower bars.
• Figure 5: Comparison of the validation loss curves.