Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model

Genís Plaja-Roglans, Yun-Ning Hung, Xavier Serra, Igor Pereira

TL;DR

This paper tackles singing voice separation from music mixtures by introducing latent diffusion in the EnCodec latent space to improve efficiency. It trains a conditioned diffusion generator on top of EnCodec embeddings, using a velocity-based objective and DDIM sampling to recover the target vocal latent conditioned on the mixture. The approach achieves competitive objective metrics and perceptual quality, outperforming generative baselines and matching non-generative systems on several measures, while enabling training with open data and faster inference. The work demonstrates the practicality of latent diffusion for music creation tasks and outlines future directions to reduce artifacts and extend to additional sources.

Abstract

Extracting individual elements from music mixtures is a valuable tool for music production and practice. While neural networks optimized to mask or transform mixture spectrograms into the individual source(s) have been the leading approach, the source overlap and correlation in music signals poses an inherent challenge. Also, accessing all sources in the mixture is crucial to train these systems, while complicated. Attempts to address these challenges in a generative fashion exist, however, the separation performance and inference efficiency remain limited. In this work, we study the potential of diffusion models to advance toward bridging this gap, focusing on generative singing voice separation relying only on corresponding pairs of isolated vocals and mixtures for training. To align with creative workflows, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio. This enables efficient optimization and faster inference. Our system is trained using only open data. We outperform existing generative separation systems, and level the compared non-generative systems on a list of signal quality measures and on interference removal. We provide a noise robustness study on the latent encoder, providing insights on its potential for the task. We release a modular toolkit for further research on the topic.

Efficient and Fast Generative-Based Singing Voice Separation using a Latent Diffusion Model

TL;DR

This paper tackles singing voice separation from music mixtures by introducing latent diffusion in the EnCodec latent space to improve efficiency. It trains a conditioned diffusion generator on top of EnCodec embeddings, using a velocity-based objective and DDIM sampling to recover the target vocal latent conditioned on the mixture. The approach achieves competitive objective metrics and perceptual quality, outperforming generative baselines and matching non-generative systems on several measures, while enabling training with open data and faster inference. The work demonstrates the practicality of latent diffusion for music creation tasks and outlines future directions to reduce artifacts and extend to additional sources.

Abstract

Extracting individual elements from music mixtures is a valuable tool for music production and practice. While neural networks optimized to mask or transform mixture spectrograms into the individual source(s) have been the leading approach, the source overlap and correlation in music signals poses an inherent challenge. Also, accessing all sources in the mixture is crucial to train these systems, while complicated. Attempts to address these challenges in a generative fashion exist, however, the separation performance and inference efficiency remain limited. In this work, we study the potential of diffusion models to advance toward bridging this gap, focusing on generative singing voice separation relying only on corresponding pairs of isolated vocals and mixtures for training. To align with creative workflows, we leverage latent diffusion: the system generates samples encoded in a compact latent space, and subsequently decodes these into audio. This enables efficient optimization and faster inference. Our system is trained using only open data. We outperform existing generative separation systems, and level the compared non-generative systems on a list of signal quality measures and on interference removal. We provide a noise robustness study on the latent encoder, providing insights on its potential for the task. We release a modular toolkit for further research on the topic.

Paper Structure

This paper contains 17 sections, 8 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Diffusion forward process
  5. Latent encoder
  6. Training algorithm
  7. Sampling process
  8. Model architecture
  9. Generator U-Net
  10. Conditioning processing and injection
  11. A two-stage training approach
  12. Experiments
  13. Experimental setup
  14. Objective evaluation
  15. Listening test
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Complete system. The boxed elements in yellow are only active during training, the rest of the system is used during training and inference. To train, we encode the target vocals and apply the forward diffusion process. The generator network is optimized to estimate a slightly denoised version of the noised target, given step $t$. The corresponding mixture is also encoded and used as conditioner. The gray dashed lines depict the inference stream, which is run for $T$ steps, to convert a Gaussian sample to the vocals contained in the conditioning mixture. Skip connections are depicted in thin, dashed arrows. Boxed in purple, the structure and nomenclature for EnCodec.
  • Figure 2: U-Net block of the generator network.