A Unified Model For Voice and Accent Conversion In Speech and Singing using Self-Supervised Learning and Feature Extraction

TL;DR

This work proposes a unified model for voice and accent conversion applicable to both speech and singing. It uses a HuBERT-based encoder to extract self-supervised linguistic/content embeddings and a HiFi-GAN decoder to synthesize the target voice, augmented by $f0$ features and singer embeddings. A two-stage training regime first leverages non-parallel data from seven singers to learn content and pitch, then fine-tunes on parallel accent data from the Speech Accent Archive to enable accent transfer while preserving content. The model optimizes with $L_{ ext{reconstruction}}$, $L_{ ext{adv}}$, and $L_{ ext{pitch}}$ losses, achieving high metrics in voice identification and accent classification compared to baselines. This framework has practical implications for voice dubbing, multilingual TTS, and IVR systems, offering natural, controllable voice transformations with preserved prosody and emotional cues.

Abstract

This paper presents a new voice conversion model capable of transforming both speaking and singing voices. It addresses key challenges in current systems, such as conveying emotions, managing pronunciation and accent changes, and reproducing non-verbal sounds. One of the model's standout features is its ability to perform accent conversion on hybrid voice samples that encompass both speech and singing, allowing it to change the speaker's accent while preserving the original content and prosody. The proposed model uses an encoder-decoder architecture: the encoder is based on HuBERT to process the speech's acoustic and linguistic content, while the HiFi-GAN decoder audio matches the target speaker's voice. The model incorporates fundamental frequency (f0) features and singer embeddings to enhance performance while ensuring the pitch & tone accuracy and vocal identity are preserved during transformation. This approach improves how naturally and flexibly voice style can be transformed, showing strong potential for applications in voice dubbing, content creation, and technologies like Text-to-Speech (TTS) and Interactive Voice Response (IVR) systems.

Figures (5)

• Figure 1: Representation of Singing Voice Conversion
• Figure 2: Representation of Accent Conversion
• Figure 3: Architecture of the proposed voice conversion model
• Figure 4: Representation of Voice and Accent Conversion Test
• Figure 5: Different Models’ Performance during Voice Conversion Test