OneVoice: One Model, Triple Scenarios-Towards Unified Zero-shot Voice Conversion
Zhichao Wang, Tao Li, Wenshuo Ge, Zihao Cui, Shilei Zhang, Junlan Feng
TL;DR
OneVoice addresses the fragmentation of voice conversion by unifying linguistic-preserving, expressive, and singing scenarios within a single, zero-shot model. It combines a continuous language model with a conditional Mixture-of-Experts to separate shared conversion knowledge from scenario-specific expressivity, guided by a dual-path routing mechanism and gated prosodic conditioning. A VAE-free next-patch diffusion backbone enables efficient, high-fidelity generation, complemented by a two-stage progressive training regime and LoRA-based domain experts to mitigate data imbalance. Empirical results show OneVoice matches or surpasses specialized models across all three VC tasks, with flexible mode switching and fast decoding, highlighting its potential for scalable, multi-scenario voice conversion in real-world applications.
Abstract
Recent progress of voice conversion~(VC) has achieved a new milestone in speaker cloning and linguistic preservation. But the field remains fragmented, relying on specialized models for linguistic-preserving, expressive, and singing scenarios. We propose OneVoice, a unified zero-shot framework capable of handling all three scenarios within a single model. OneVoice is built upon a continuous language model trained with VAE-free next-patch diffusion, ensuring high fidelity and efficient sequence modeling. Its core design for unification lies in a Mixture-of-Experts (MoE) designed to explicitly model shared conversion knowledge and scenario-specific expressivity. Expert selection is coordinated by a dual-path routing mechanism, including shared expert isolation and scenario-aware domain expert assignment with global-local cues. For precise conditioning, scenario-specific prosodic features are fused into each layer via a gated mechanism, allowing adaptive usage of prosody information. Furthermore, to enable the core idea and alleviate the imbalanced issue (abundant speech vs. scarce singing), we adopt a two-stage progressive training that includes foundational pre-training and scenario enhancement with LoRA-based domain experts. Experiments show that OneVoice matches or surpasses specialized models across all three scenarios, while verifying flexible control over scenarios and offering a fast decoding version as few as 2 steps. Code and model will be released soon.