Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EELE: Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech

Xin Qi, Ruibo Fu, Zhengqi Wen, Jianhua Tao, Shuchen Shi, Yi Lu, Zhiyong Wang, Xiaopeng Wang, Yuankun Xie, Yukun Liu, Guanjun Li, Xuefei Liu, Yongwei Li

TL;DR

This work tackles the scalability and flexibility challenge of endowing TTS with emotions by introducing EELE, a plug-and-play approach that starts from a neutral TTS model and adds emotion capability post-training via LoRA adapters. By experimenting with eight module-level LoRA deployment schemes in the VITS2 backbone, the authors systematically evaluate where emotion should be modeled and how duration cues influence expression. Key findings show that certain LoRA placements (notably scheme 'g') yield superior emotion recognition, with anger most prominent and surprise the hardest to detect, while smaller LoRA ranks can achieve competitive performance, offering significant efficiency over full fine-tuning. The approach demonstrates practical advantages in adaptability and scalability for emotion-rich TTS across applications like media and gaming.

Abstract

In the current era of Artificial Intelligence Generated Content (AIGC), a Low-Rank Adaptation (LoRA) method has emerged. It uses a plugin-based approach to learn new knowledge with lower parameter quantities and computational costs, and it can be plugged in and out based on the specific sub-tasks, offering high flexibility. However, the current application schemes primarily incorporate LoRA into the pre-introduced conditional parts of the speech models. This fixes the position of LoRA, limiting the flexibility and scalability of its application. Therefore, we propose the Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech (EELE) method. Starting from a general neutral speech model, we do not pre-introduce emotional information but instead use the LoRA plugin to design a flexible adaptive scheme that endows the model with emotional generation capabilities. Specifically, we initially train the model using only neutral speech data. After training is complete, we insert LoRA into different modules and fine-tune the model with emotional speech data to find the optimal insertion scheme. Through experiments, we compare and test the effects of inserting LoRA at different positions within the model and assess LoRA's ability to learn various emotions, effectively proving the validity of our method. Additionally, we explore the impact of the rank size of LoRA and the difference compared to directly fine-tuning the entire model.

EELE: Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech

TL;DR

This work tackles the scalability and flexibility challenge of endowing TTS with emotions by introducing EELE, a plug-and-play approach that starts from a neutral TTS model and adds emotion capability post-training via LoRA adapters. By experimenting with eight module-level LoRA deployment schemes in the VITS2 backbone, the authors systematically evaluate where emotion should be modeled and how duration cues influence expression. Key findings show that certain LoRA placements (notably scheme 'g') yield superior emotion recognition, with anger most prominent and surprise the hardest to detect, while smaller LoRA ranks can achieve competitive performance, offering significant efficiency over full fine-tuning. The approach demonstrates practical advantages in adaptability and scalability for emotion-rich TTS across applications like media and gaming.

Abstract

In the current era of Artificial Intelligence Generated Content (AIGC), a Low-Rank Adaptation (LoRA) method has emerged. It uses a plugin-based approach to learn new knowledge with lower parameter quantities and computational costs, and it can be plugged in and out based on the specific sub-tasks, offering high flexibility. However, the current application schemes primarily incorporate LoRA into the pre-introduced conditional parts of the speech models. This fixes the position of LoRA, limiting the flexibility and scalability of its application. Therefore, we propose the Exploring Efficient and Extensible LoRA Integration in Emotional Text-to-Speech (EELE) method. Starting from a general neutral speech model, we do not pre-introduce emotional information but instead use the LoRA plugin to design a flexible adaptive scheme that endows the model with emotional generation capabilities. Specifically, we initially train the model using only neutral speech data. After training is complete, we insert LoRA into different modules and fine-tune the model with emotional speech data to find the optimal insertion scheme. Through experiments, we compare and test the effects of inserting LoRA at different positions within the model and assess LoRA's ability to learn various emotions, effectively proving the validity of our method. Additionally, we explore the impact of the rank size of LoRA and the difference compared to directly fine-tuning the entire model.
Paper Structure (16 sections, 2 figures, 4 tables)

This paper contains 16 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Where to add: Exploring the deployment of LoRA
  4. How to add: Implantation of LORA in TTS model
  5. Experiment Setup
  6. DataSet
  7. Metric
  8. Task
  9. Experiment and Analysis
  10. Subjective evaluation
  11. Objective evaluation
  12. Emotion recognition results
  13. The impact of changes in r-value
  14. Compared with fine-tuning
  15. Conclusion
  16. ...and 1 more sections

Figures (2)

  • Figure 1: An overview of all attempts to add LoRA to the model. represents freezing all parameters. represents freezing all parameters and adding LoRA for fine-tuning.
  • Figure 2: LoRA adds low-rank matrices to each layer.In the diagram, the blue $W$ represents the frozen original model parameters, the red $A$ and $B$ represent the low-rank matrices, and the yellow $x$ and $h$ represent the layer's input and output, respectively.