Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation

Shihao Chen, Yu Gu, Jianwei Cui, Jie Zhang, Rilin Chen, Lirong Dai

TL;DR

This work tackles the latency of diffusion-based singing voice conversion by introducing LCM-SVC, which leverages latent consistency distillation to convert a pre-trained latent diffusion model into a fast, one-step or few-step generator without sacrificing timbre decoupling or sound quality. The method first trains a So-VITS-SVC-based latent encoder–decoder and a teacher LDM, then performs LCD to learn a consistent mapping in latent space, enabling skipping and EMA-based teacher updates to achieve dramatic speedups. Experimental results on OpenSinger show that 2-step or 4-step LCD delivers performance close to the teacher with far lower real-time factors, while 1-step inference incurs a modest quality drop; the approach thus offers a practical trade-off between latency and fidelity for real-time SVC. Overall, LCM-SVC provides a scalable path to high-quality, fast any-to-any SVC by distilling diffusion models in latent space, with potential impact on real-time audio editing and vocal timbre transfer.

Abstract

Any-to-any singing voice conversion (SVC) aims to transfer a target singer's timbre to other songs using a short voice sample. However many diffusion model based any-to-any SVC methods, which have achieved impressive results, usually suffered from low efficiency caused by a mass of inference steps. In this paper, we propose LCM-SVC, a latent consistency distillation (LCD) based latent diffusion model (LDM) to accelerate inference speed. We achieved one-step or few-step inference while maintaining the high performance by distilling a pre-trained LDM based SVC model, which had the advantages of timbre decoupling and sound quality. Experimental results show that our proposed method can significantly reduce the inference time and largely preserve the sound quality and timbre similarity comparing with other state-of-the-art SVC models. Audio samples are available at https://sounddemos.github.io/lcm-svc.

LCM-SVC: Latent Diffusion Model Based Singing Voice Conversion with Inference Acceleration via Latent Consistency Distillation

TL;DR

This work tackles the latency of diffusion-based singing voice conversion by introducing LCM-SVC, which leverages latent consistency distillation to convert a pre-trained latent diffusion model into a fast, one-step or few-step generator without sacrificing timbre decoupling or sound quality. The method first trains a So-VITS-SVC-based latent encoder–decoder and a teacher LDM, then performs LCD to learn a consistent mapping in latent space, enabling skipping and EMA-based teacher updates to achieve dramatic speedups. Experimental results on OpenSinger show that 2-step or 4-step LCD delivers performance close to the teacher with far lower real-time factors, while 1-step inference incurs a modest quality drop; the approach thus offers a practical trade-off between latency and fidelity for real-time SVC. Overall, LCM-SVC provides a scalable path to high-quality, fast any-to-any SVC by distilling diffusion models in latent space, with potential impact on real-time audio editing and vocal timbre transfer.

Abstract

Any-to-any singing voice conversion (SVC) aims to transfer a target singer's timbre to other songs using a short voice sample. However many diffusion model based any-to-any SVC methods, which have achieved impressive results, usually suffered from low efficiency caused by a mass of inference steps. In this paper, we propose LCM-SVC, a latent consistency distillation (LCD) based latent diffusion model (LDM) to accelerate inference speed. We achieved one-step or few-step inference while maintaining the high performance by distilling a pre-trained LDM based SVC model, which had the advantages of timbre decoupling and sound quality. Experimental results show that our proposed method can significantly reduce the inference time and largely preserve the sound quality and timbre similarity comparing with other state-of-the-art SVC models. Audio samples are available at https://sounddemos.github.io/lcm-svc.
Paper Structure (9 sections, 6 equations, 2 figures, 3 tables, 2 algorithms)

This paper contains 9 sections, 6 equations, 2 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Method
  3. Latent Diffusion Model for SVC
  4. Latent Consistency Distillation
  5. Experiment
  6. Experimental Setup
  7. Evaluation Metrics
  8. Experimental Results
  9. Conclusion

Figures (2)

  • Figure 1: Left: Pre-training procedure of So-VITS-SVC; Right: Training procedure of LCM-SVC Teacher.
  • Figure 2: The Training and Inference procedure of LCD.