DualVC 2: Dynamic Masked Convolution for Unified Streaming and Non-Streaming Voice Conversion

TL;DR

This work tackles real-time voice conversion by addressing limitations in prior streaming models, notably autoregressive decoding latency, limited future-context within chunks, and unvoiced-frame noise. It introduces DualVC 2, a Conformer-based recognition-synthesis framework that uses dynamic chunk training, dynamic masked convolution, quiet attention, and data augmentation, while leveraging hybrid predictive coding. The approach yields a streaming VC system with a pipeline latency of 186.4 ms and an overall real-time factor of $0.165$, outperforming DualVC and IBF-VC on naturalness, speaker similarity, and robustness to noise. The results demonstrate the practical viability of high-quality, low-latency streaming VC for real-time applications such as live communication and streaming.

Abstract

Voice conversion is becoming increasingly popular, and a growing number of application scenarios require models with streaming inference capabilities. The recently proposed DualVC attempts to achieve this objective through streaming model architecture design and intra-model knowledge distillation along with hybrid predictive coding to compensate for the lack of future information. However, DualVC encounters several problems that limit its performance. First, the autoregressive decoder has error accumulation in its nature and limits the inference speed as well. Second, the causal convolution enables streaming capability but cannot sufficiently use future information within chunks. Third, the model is unable to effectively address the noise in the unvoiced segments, lowering the sound quality. In this paper, we propose DualVC 2 to address these issues. Specifically, the model backbone is migrated to a Conformer-based architecture, empowering parallel inference. Causal convolution is replaced by non-causal convolution with dynamic chunk mask to make better use of within-chunk future information. Also, quiet attention is introduced to enhance the model's noise robustness. Experiments show that DualVC 2 outperforms DualVC and other baseline systems in both subjective and objective metrics, with only 186.4 ms latency. Our audio samples are made publicly available.

Figures (3)

• Figure 1: The architecture of DualVC 2.
• Figure 2: Dynamic Masked Convolution Module. Non-causal 1D convolution is replaced by equivalent 2D convolution. A dynamic mask is applied to simulate different lengths of future frames in the convolutional kernel receptive field.
• Figure 3: Visualizations of generated mel-spectrograms by DualVC 2 and ablation systems.