Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion

Zhichao Wang, Yuanzhe Chen, Xinsheng Wang, Lei Xie, Yuping Wang

TL;DR

StreamVoice presents a fully causal, streaming LM-based framework for zero-shot voice conversion that alternates semantic and acoustic inputs and introduces context-aware enhancements to mitigate incomplete streaming context. By combining a decoder-only LM with an acoustic predictor and employing semantic masking along with teacher-guided context foresight, the method achieves real-time conversion with latency around $124$ ms while maintaining competitive zero-shot and in-dataset performance without any future look-ahead. The approach demonstrates that LM-based streaming VC can reach practical real-time speeds (RTF $<1$) on standard GPUs and closely match non-streaming baselines, significantly advancing the applicability of zero-shot VC in live settings. The work also provides analyses of ASR/codec dependencies and ablations that validate the effectiveness of context-aware strategies for robust streaming voice conversion.

Abstract

Recent language model (LM) advancements have showcased impressive zero-shot voice conversion (VC) performance. However, existing LM-based VC models usually apply offline conversion from source semantics to acoustic features, demanding the complete source speech and limiting their deployment to real-time applications. In this paper, we introduce StreamVoice, a novel streaming LM-based model for zero-shot VC, facilitating real-time conversion given arbitrary speaker prompts and source speech. Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech. To address the potential performance degradation from the incomplete context in streaming processing, we enhance the context-awareness of the LM through two strategies: 1) teacher-guided context foresight, using a teacher model to summarize the present and future semantic context during training to guide the model's forecasting for missing context; 2) semantic masking strategy, promoting acoustic prediction from preceding corrupted semantic and acoustic input, enhancing context-learning ability. Notably, StreamVoice is the first LM-based streaming zero-shot VC model without any future look-ahead. Experiments demonstrate StreamVoice's streaming conversion capability while achieving zero-shot performance comparable to non-streaming VC systems.

StreamVoice: Streamable Context-Aware Language Modeling for Real-time Zero-Shot Voice Conversion

TL;DR

StreamVoice presents a fully causal, streaming LM-based framework for zero-shot voice conversion that alternates semantic and acoustic inputs and introduces context-aware enhancements to mitigate incomplete streaming context. By combining a decoder-only LM with an acoustic predictor and employing semantic masking along with teacher-guided context foresight, the method achieves real-time conversion with latency around ms while maintaining competitive zero-shot and in-dataset performance without any future look-ahead. The approach demonstrates that LM-based streaming VC can reach practical real-time speeds (RTF ) on standard GPUs and closely match non-streaming baselines, significantly advancing the applicability of zero-shot VC in live settings. The work also provides analyses of ASR/codec dependencies and ablations that validate the effectiveness of context-aware strategies for robust streaming voice conversion.

Abstract

Recent language model (LM) advancements have showcased impressive zero-shot voice conversion (VC) performance. However, existing LM-based VC models usually apply offline conversion from source semantics to acoustic features, demanding the complete source speech and limiting their deployment to real-time applications. In this paper, we introduce StreamVoice, a novel streaming LM-based model for zero-shot VC, facilitating real-time conversion given arbitrary speaker prompts and source speech. Specifically, to enable streaming capability, StreamVoice employs a fully causal context-aware LM with a temporal-independent acoustic predictor, while alternately processing semantic and acoustic features at each time step of autoregression which eliminates the dependence on complete source speech. To address the potential performance degradation from the incomplete context in streaming processing, we enhance the context-awareness of the LM through two strategies: 1) teacher-guided context foresight, using a teacher model to summarize the present and future semantic context during training to guide the model's forecasting for missing context; 2) semantic masking strategy, promoting acoustic prediction from preceding corrupted semantic and acoustic input, enhancing context-learning ability. Notably, StreamVoice is the first LM-based streaming zero-shot VC model without any future look-ahead. Experiments demonstrate StreamVoice's streaming conversion capability while achieving zero-shot performance comparable to non-streaming VC systems.
Paper Structure (19 sections, 3 equations, 4 figures, 6 tables)

This paper contains 19 sections, 3 equations, 4 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. StreamVoice
  4. Overview
  5. Streamable Architecture
  6. Fully Casual Language Model
  7. Acoustic Predictor
  8. Context-aware Enhancement
  9. Training & Inference Procedure
  10. Experiments
  11. Experimental Setup
  12. Experiments Results
  13. Zero-shot Evaluation
  14. In-dataset Evaluation
  15. Ablation Study
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The concept of the streaming zero-shot VC employing the widely used recognition-synthesis framework PPGSun2016PhoneticPF, where only the encoder of ASR is involved. StreamVoice is built on this popular paradigm.
  • Figure 2: The overall architecture for StreamVoice.
  • Figure 3: The architecture for context-aware LM.
  • Figure 4: The architecture for acoustic predictor. Our system can support continuous or discrete projection.