Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning

Chung-Ming Chien, Andros Tjandra, Apoorv Vyas, Matt Le, Bowen Shi, Wei-Ning Hsu

TL;DR

This work tackles the challenge of injecting fine-grained controllability into a pre-trained speech generation model without full fine-tuning. It introduces Voicebox Adapter, which uses cross-attention and parameter-efficient adapters (notably LoRA with bias-tuning) to incorporate fine-grained conditions such as punctuation, emphasis, and laughter, achieving performance close to full-model fine-tuning with a small fraction of trainable parameters. Empirical results show robust improvements in fine-grained control, intelligibility, and speaker similarity across three tasks, though laughter remains data-sensitive due to limited pre-training coverage (only about $1.4\%$ of frames). The approach offers a practical, scalable path to deploy finely controlled speech generation in large models, with strong cross-task generalization and clear guidance on effective fine-tuning strategies.

Abstract

As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups.

Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning

TL;DR

This work tackles the challenge of injecting fine-grained controllability into a pre-trained speech generation model without full fine-tuning. It introduces Voicebox Adapter, which uses cross-attention and parameter-efficient adapters (notably LoRA with bias-tuning) to incorporate fine-grained conditions such as punctuation, emphasis, and laughter, achieving performance close to full-model fine-tuning with a small fraction of trainable parameters. Empirical results show robust improvements in fine-grained control, intelligibility, and speaker similarity across three tasks, though laughter remains data-sensitive due to limited pre-training coverage (only about of frames). The approach offers a practical, scalable path to deploy finely controlled speech generation in large models, with strong cross-task generalization and clear guidance on effective fine-tuning strategies.

Abstract

As the scale of generative models continues to grow, efficient reuse and adaptation of pre-trained models have become crucial considerations. In this work, we propose Voicebox Adapter, a novel approach that integrates fine-grained conditions into a pre-trained Voicebox speech generation model using a cross-attention module. To ensure a smooth integration of newly added modules with pre-trained ones, we explore various efficient fine-tuning approaches. Our experiment shows that the LoRA with bias-tuning configuration yields the best performance, enhancing controllability without compromising speech quality. Across three fine-grained conditional generation tasks, we demonstrate the effectiveness and resource efficiency of Voicebox Adapter. Follow-up experiments further highlight the robustness of Voicebox Adapter across diverse data setups.
Paper Structure (21 sections, 4 equations, 2 figures, 2 tables)

This paper contains 21 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Adaptive fine-grained controllable speech generation
  4. Efficient fine-tuning for Transformers
  5. Background
  6. Proposed method
  7. Voicebox Adapter
  8. Efficient fine-tuning methods
  9. Experimental setup
  10. Pre-training details
  11. Fine-grained conditional fine-tuning setup
  12. Inference setup
  13. Objective evaluation methods
  14. Fine-grained controllability
  15. Intelligibility and speaker similarity
  16. ...and 6 more sections

Figures (2)

  • Figure 1: The model architecture of Voicebox Adapter, and a zoom-in view of a Transformer layer.
  • Figure 2: Performance of Voicebox Adapter and compared models with different hidden dimensions and data configurations.