Characteristic-Specific Partial Fine-Tuning for Efficient Emotion and Speaker Adaptation in Codec Language Text-to-Speech Models

TL;DR

Experimental results demonstrate that the proposed characteristic-specific partial fine-tuning strategy, short as CSP-FT, achieves performance comparable to, or even surpassing, full fine-tuning in generating speech with specific emotional expressions and speaker identities.

Abstract

Recently, emotional speech generation and speaker cloning have garnered significant interest in text-to-speech (TTS). With the open-sourcing of codec language TTS models trained on massive datasets with large-scale parameters, adapting these general pre-trained TTS models to generate speech with specific emotional expressions and target speaker characteristics has become a topic of great attention. Common approaches, such as full and adapter-based fine-tuning, often overlook the specific contributions of model parameters to emotion and speaker control. Treating all parameters uniformly during fine-tuning, especially when the target data has limited content diversity compared to the pre-training corpus, results in slow training speed and an increased risk of catastrophic forgetting. To address these challenges, we propose a characteristic-specific partial fine-tuning strategy, short as CSP-FT. First, we use a weighted-sum approach to analyze the contributions of different Transformer layers in a pre-trained codec language TTS model for emotion and speaker control in the generated speech. We then selectively fine-tune the layers with the highest and lowest characteristic-specific contributions to generate speech with target emotional expression and speaker identity. Experimental results demonstrate that our method achieves performance comparable to, or even surpassing, full fine-tuning in generating speech with specific emotional expressions and speaker identities. Additionally, CSP-FT delivers approximately 2x faster training speeds, fine-tunes only around 8% of parameters, and significantly reduces catastrophic forgetting. Furthermore, we show that codec language TTS models perform competitively with self-supervised models in speaker identification and emotion classification tasks, offering valuable insights for developing universal speech processing models.

Figures (7)

• Figure 1: Characteristic-specific partial fine-tuning for emotion and speaker adaptation in codec language TTS models.
• Figure 2: Overview of the proposed approach. The codec language TTS model is fine-tuned for emotion recognition and speaker identification tasks using a weighted-sum framework. The method prioritizes layers with the lowest and highest weighted importance, facilitating efficient fine-tuning to generate speech with target-domain specific speaker and emotional expressions.
• Figure 3: Single-Layer fine-tuning performance and layer-wise characteristic-specific weights for emotion and speaker adaptation GPT-SoVITS.
• Figure 4: Single-Layer fine-tuning performance and layer-wise characteristic-specific weights for emotion and speaker adaptation in VALLE-X.
• Figure 5: Single-Layer fine-tuning performance and layer-wise characteristic-specific weights for emotion and speaker adaptation CosyVoice.