Stepback: Enhanced Disentanglement for Voice Conversion via Multi-Task Learning

TL;DR

The paper tackles non-parallel voice conversion by focusing on disentangling linguistic content from speaker identity. It introduces the Stepback framework, combining a Preparatory Stage that stabilizes an encoder–decoder with a Proposed Stage that employs dual-domain data and two decoders guided by multi-task losses, including a self-destructive amendment mechanism formalized as $L_{back} = -\lambda L_{upp} + L_{low}$, to enhance content preservation while suppressing residual speaker cues. Through experiments on the VCTK dataset, the approach achieves comparable or improved voice conversion quality at roughly 70% of the training cost of a benchmark model, aided by a classifier-guided constraint and GAN-based refinement. The results suggest Stepback as a scalable strategy for robust, data-efficient VC in non-parallel settings, with potential extensions to advanced speaker representations.

Abstract

Voice conversion (VC) modifies voice characteristics while preserving linguistic content. This paper presents the Stepback network, a novel model for converting speaker identity using non-parallel data. Unlike traditional VC methods that rely on parallel data, our approach leverages deep learning techniques to enhance disentanglement completion and linguistic content preservation. The Stepback network incorporates a dual flow of different domain data inputs and uses constraints with self-destructive amendments to optimize the content encoder. Extensive experiments show that our model significantly improves VC performance, reducing training costs while achieving high-quality voice conversion. The Stepback network's design offers a promising solution for advanced voice conversion tasks.

Figures (1)

• Figure 1: Basic Schematic diagram of mini-stage 2.