Optimal Transport Maps are Good Voice Converters

TL;DR

This paper presents a variety of optimal transport algorithms designed for different data representations, such as mel-spectrograms and latent representation of self-supervised speech models, and achieves strong results in terms of Frechet Audio Distance.

Abstract

Recently, neural network-based methods for computing optimal transport maps have been effectively applied to style transfer problems. However, the application of these methods to voice conversion is underexplored. In our paper, we fill this gap by investigating optimal transport as a framework for voice conversion. We present a variety of optimal transport algorithms designed for different data representations, such as mel-spectrograms and latent representation of self-supervised speech models. For the mel-spectogram data representation, we achieve strong results in terms of Frechet Audio Distance (FAD). This performance is consistent with our theoretical analysis, which suggests that our method provides an upper bound on the FAD between the target and generated distributions. Within the latent space of the WavLM encoder, we achived state-of-the-art results and outperformed existing methods even with limited reference speaker data.

Figures (7)

• Figure 1: First, the mel-spectrogram of the source speaker is fed into the map $T$, while the reference is fed into the speaker encoder and also used as input for $T$ and $f$ (during training). The map $T$ outputs the converted spectrogram, which is transformed back into raw audio by the vocoder..
• Figure 2: Wav audio is fed into the WavLM. At the same time, the reference is fed into the WavLM. Then the OT matching $\mathcal{L}_{OTM}$ (SinkVC or FMVC) converts the voice into the given latent representations. After inference, the results are transformed back into raw audio using the vocoder.
• Figure 3: The figure displays the WER score in relation to the size of the target speaker's speech.
• Figure 4: FAD scores for the many-to-many conversion problem, during training of our proposed XNOT-VC method. Steps are in 10e4 scale.
• Figure 5: pMOS scores in depending of the provided target len.