Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards General-Purpose Text-Instruction-Guided Voice Conversion

Chun-Yi Kuan, Chen An Li, Tsu-Yuan Hsu, Tse-Yang Lin, Ho-Lam Chung, Kai-Wei Chang, Shuo-yiin Chang, Hung-yi Lee

TL;DR

This work introduces a text-instruction-guided voice conversion framework that operates without relying on style reference utterances. It treats voice conversion as conditional codec language modeling, encoding source speech into discrete tokens via EnCodec and generating target codes with autoregressive and non-autoregressive models conditioned on natural language instructions. The authors explore pretraining regimes (Text and TTS) and construct two datasets—the Signal Processing Effect Dataset and InstructSpeech Dataset—to enable instruction-based style control and evaluation of generalization to unseen instructions, including varying adverbs. Results show strong instruction comprehension and capable style transformation, though further work is needed to improve speech quality and diversify instruction types for broader applicability.

Abstract

This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice". Unlike traditional methods that rely on reference utterances to determine the attributes of the converted speech, our model adds versatility and specificity to voice conversion. The proposed VC model is a neural codec language model which processes a sequence of discrete codes, resulting in the code sequence of converted speech. It utilizes text instructions as style prompts to modify the prosody and emotional information of the given speech. In contrast to previous approaches, which often rely on employing separate encoders like prosody and content encoders to handle different aspects of the source speech, our model handles various information of speech in an end-to-end manner. Experiments have demonstrated the impressive capabilities of our model in comprehending instructions and delivering reasonable results.

Towards General-Purpose Text-Instruction-Guided Voice Conversion

TL;DR

This work introduces a text-instruction-guided voice conversion framework that operates without relying on style reference utterances. It treats voice conversion as conditional codec language modeling, encoding source speech into discrete tokens via EnCodec and generating target codes with autoregressive and non-autoregressive models conditioned on natural language instructions. The authors explore pretraining regimes (Text and TTS) and construct two datasets—the Signal Processing Effect Dataset and InstructSpeech Dataset—to enable instruction-based style control and evaluation of generalization to unseen instructions, including varying adverbs. Results show strong instruction comprehension and capable style transformation, though further work is needed to improve speech quality and diversify instruction types for broader applicability.

Abstract

This paper introduces a novel voice conversion (VC) model, guided by text instructions such as "articulate slowly with a deep tone" or "speak in a cheerful boyish voice". Unlike traditional methods that rely on reference utterances to determine the attributes of the converted speech, our model adds versatility and specificity to voice conversion. The proposed VC model is a neural codec language model which processes a sequence of discrete codes, resulting in the code sequence of converted speech. It utilizes text instructions as style prompts to modify the prosody and emotional information of the given speech. In contrast to previous approaches, which often rely on employing separate encoders like prosody and content encoders to handle different aspects of the source speech, our model handles various information of speech in an end-to-end manner. Experiments have demonstrated the impressive capabilities of our model in comprehending instructions and delivering reasonable results.
Paper Structure (20 sections, 3 figures, 3 tables)

This paper contains 20 sections, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Codec Model
  4. Textual-guided speech generation
  5. Style conversion
  6. Method
  7. Problem formulation
  8. Training: Conditional Codec Language Modeling
  9. Pre-training
  10. Inference
  11. Dataset
  12. Overview
  13. Signal Processing Effect Dataset
  14. InstructSpeech Dataset
  15. Experiment
  16. ...and 5 more sections

Figures (3)

  • Figure 1: The overview of our framework. Conditioned on the given textual instruction and source speech, our model is able to generate sequences of discrete audio codes. Subsequently, we decode these sequences of codes to the corresponding target speech by audio codec decoder. In addition, we propose two pre-training methods. Method 1 involves fine-tuning the textual language model. Method 2 involves fine-tuning the text-to-speech pre-trained language model.
  • Figure 2: The architecture of autoregressive and non-autoregressive codec language model.
  • Figure 3: From top to bottom, (1) Spectrogram of the source speech (2) Spectrogram of the target speech with the instruction as "Decrease the speed of speech slightly." (3) "Decrease the speed of speech." (4) "Decrease the speed of speech notably." (5) "Decrease the speed of speech extremely." The length of the audio, originally 2.47 seconds in the source speech, is extended to 3.17, 5.25, 8.44, and 10.27 seconds respectively.