Discrete Optimal Transport and Voice Conversion
Anton Selitskiy, Maitreya Kocharekar
TL;DR
The paper tackles voice conversion by casting embedding alignment as a discrete optimal transport problem and mapping source to target embeddings via barycentric projection. It shows that OT-based mappings, especially the barycentric approach (OT-BAR), outperform simple kNN-averaging, and that longer target utterances improve conversion quality. A notable contribution is demonstrating cross-domain adaptation by applying OT to spoofed audio, significantly challenging spoof detectors like AASIST, which highlights potential security implications. Overall, the work provides a practical OT-based VC pipeline with robust ablations and reveals important insights into duration effects and domain alignment in audio generation.
Abstract
In this work, we address the voice conversion (VC) task using a vector-based interface. To align audio embeddings between speakers, we employ discrete optimal transport mapping. Our evaluation results demonstrate the high quality and effectiveness of this method. Additionally, we show that applying discrete optimal transport as a post-processing step in audio generation can lead to the incorrect classification of synthetic audio as real.