Multi-level Temporal-channel Speaker Retrieval for Zero-shot Voice Conversion

TL;DR

This work tackles zero-shot voice conversion by modeling speaker timbre as a dynamic property that varies across temporal and spectral (channel) dimensions. It introduces MTCR-VC, which uses multi-level temporal-channel retrieval (MTCR) with stacked TCR blocks to flexibly retrieve speaker representations guided by an off-the-shelf x-vector, enabling fine-grained timbre transfer. A cycle-based training strategy with paired and unpaired paths plus perceptual constraints on content, style, and speaker improves disentanglement and reconstruction under zero-shot conditions. Experimental results show MTCR-VC achieves superior speaker similarity and competitive naturalness compared with state-of-the-art zero-shot VC models, demonstrating robustness to unseen speakers and recording conditions. The approach advances zero-shot VC by enabling time- and frequency-aware speaker modeling, with implications for more flexible and scalable voice conversion systems.

Abstract

Zero-shot voice conversion (VC) converts source speech into the voice of any desired speaker using only one utterance of the speaker without requiring additional model updates. Typical methods use a speaker representation from a pre-trained speaker verification (SV) model or learn speaker representation during VC training to achieve zero-shot VC. However, existing speaker modeling methods overlook the variation of speaker information richness in temporal and frequency channel dimensions of speech. This insufficient speaker modeling hampers the ability of the VC model to accurately represent unseen speakers who are not in the training dataset. In this study, we present a robust zero-shot VC model with multi-level temporal-channel retrieval, referred to as MTCR-VC. Specifically, to flexibly adapt to the dynamic-variant speaker characteristic in the temporal and channel axis of the speech, we propose a novel fine-grained speaker modeling method, called temporal-channel retrieval (TCR), to find out when and where speaker information appears in speech. It retrieves variable-length speaker representation from both temporal and channel dimensions under the guidance of a pre-trained SV model. Besides, inspired by the hierarchical process of human speech production, the MTCR speaker module stacks several TCR blocks to extract speaker representations from multi-granularity levels. Furthermore, to achieve better speech disentanglement and reconstruction, we introduce a cycle-based training strategy to simulate zero-shot inference recurrently. We adopt perpetual constraints on three aspects, including content, style, and speaker, to drive this process. Experiments demonstrate that MTCR-VC is superior to the previous zero-shot VC methods in modeling speaker timbre while maintaining good speech naturalness.

Figures (9)

• Figure 1: The typical frameworks of zero-shot VC. Speaker modeling can be at (a) utterance-level or fine-grained-level with two types: time-varying (b) and multi-level fusion (c).
• Figure 2: The framework of the proposed MTCR-VC model.
• Figure 3: The architecture of the TCR block. Note that the symbols in red color represent the outputs of the block.
• Figure 4: The (a) fusion block in the speech decoder. The symbols in red color represent the outputs of the fusion block. The architecture of the fusion block follows the design of Extractor FragmentvcAVLin2021.
• Figure 5: The cycle-based training process of MTCR-VC. (a) Paired Path. (b) Unpaired Path. Note that speech utterances X and Y come from different speakers with different linguistic content.