Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

StreamVC: Real-Time Low-Latency Voice Conversion

Yang Yang, Yury Kartynnik, Yunpeng Li, Jiuqiang Tang, Xing Li, George Sung, Matthias Grundmann

TL;DR

StreamVC addresses real-time, on-device voice conversion that preserves content while adopting target timbre. It employs a SoundStream-inspired neural audio codec with a causal content encoder trained on HuBERT-derived soft speech units, a separate speaker encoder, and a FiLM-conditioned decoder. The approach injects whitened $f_0$ and frame energy to stabilize pitch without leaking source timbre, enabling streaming inference with an end-to-end latency of $70.8$ ms on a Pixel 7. Experiments show competitive naturalness, intelligibility, and speaker similarity, with high $f_0$ consistency and notable gains from finetuning on the target domain; ablations confirm the necessity of $f_0$ whitening and $f_0$ conditioning for robust pitch and identity preservation.

Abstract

We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information.

StreamVC: Real-Time Low-Latency Voice Conversion

TL;DR

StreamVC addresses real-time, on-device voice conversion that preserves content while adopting target timbre. It employs a SoundStream-inspired neural audio codec with a causal content encoder trained on HuBERT-derived soft speech units, a separate speaker encoder, and a FiLM-conditioned decoder. The approach injects whitened and frame energy to stabilize pitch without leaking source timbre, enabling streaming inference with an end-to-end latency of ms on a Pixel 7. Experiments show competitive naturalness, intelligibility, and speaker similarity, with high consistency and notable gains from finetuning on the target domain; ablations confirm the necessity of whitening and conditioning for robust pitch and identity preservation.

Abstract

We present StreamVC, a streaming voice conversion solution that preserves the content and prosody of any source speech while matching the voice timbre from any target speech. Unlike previous approaches, StreamVC produces the resulting waveform at low latency from the input signal even on a mobile platform, making it applicable to real-time communication scenarios like calls and video conferencing, and addressing use cases such as voice anonymization in these scenarios. Our design leverages the architecture and training strategy of the SoundStream neural audio codec for lightweight high-quality speech synthesis. We demonstrate the feasibility of learning soft speech units causally, as well as the effectiveness of supplying whitened fundamental frequency information to improve pitch stability without leaking the source timbre information.
Paper Structure (22 sections, 2 figures, 1 table)

This paper contains 22 sections, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method
  3. Motivation
  4. Model Architecture
  5. Content Encoder
  6. Speaker Encoder
  7. Fundamental Frequency and Energy Estimation
  8. Decoder
  9. Training Strategy
  10. Soft-Label Creation and Content Encoder Training
  11. Training Loss
  12. Real-Time Inference
  13. Streaming Inference
  14. Lookahead and Architectural Latency
  15. Computational Latency
  16. ...and 7 more sections

Figures (2)

  • Figure 1: System diagram of StreamVC.
  • Figure 2: Streaming inference.