Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework

Hokuto Munakata, Ryo Terashima, Yusuke Fujita

TL;DR

A data cleansing method that utilizes a neural analysis and synthesis (NANSY++) framework to train an end-to-end neural diarization model (EEND) for singer diarization that mitigates the mislabeling of choral singing to solo singing and helps the effective training of EEND models even when the majority of available song data contains choral singing sections.

Abstract

We propose a data cleansing method that utilizes a neural analysis and synthesis (NANSY++) framework to train an end-to-end neural diarization model (EEND) for singer diarization. Our proposed model converts song data with choral singing which is commonly contained in popular music and unsuitable for generating a simulated dataset to the solo singing data. This cleansing is based on NANSY++, which is a framework trained to reconstruct an input non-overlapped audio signal. We exploit the pre-trained NANSY++ to convert choral singing into clean, non-overlapped audio. This cleansing process mitigates the mislabeling of choral singing to solo singing and helps the effective training of EEND models even when the majority of available song data contains choral singing sections. We experimentally evaluated the EEND model trained with a dataset using our proposed method using annotated popular duet songs. As a result, our proposed method improved 14.8 points in diarization error rate.

Song Data Cleansing for End-to-End Neural Singer Diarization Using Neural Analysis and Synthesis Framework

TL;DR

A data cleansing method that utilizes a neural analysis and synthesis (NANSY++) framework to train an end-to-end neural diarization model (EEND) for singer diarization that mitigates the mislabeling of choral singing to solo singing and helps the effective training of EEND models even when the majority of available song data contains choral singing sections.

Abstract

We propose a data cleansing method that utilizes a neural analysis and synthesis (NANSY++) framework to train an end-to-end neural diarization model (EEND) for singer diarization. Our proposed model converts song data with choral singing which is commonly contained in popular music and unsuitable for generating a simulated dataset to the solo singing data. This cleansing is based on NANSY++, which is a framework trained to reconstruct an input non-overlapped audio signal. We exploit the pre-trained NANSY++ to convert choral singing into clean, non-overlapped audio. This cleansing process mitigates the mislabeling of choral singing to solo singing and helps the effective training of EEND models even when the majority of available song data contains choral singing sections. We experimentally evaluated the EEND model trained with a dataset using our proposed method using annotated popular duet songs. As a result, our proposed method improved 14.8 points in diarization error rate.

Paper Structure

This paper contains 15 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. End-to-End Neural Diarization
  4. Neural Analysis and Synthesis Framework
  5. Proposed method
  6. Data Cleansing Using NANSY++
  7. Training Data Generation
  8. Experimental Evaluation
  9. Dataset
  10. Evaluation
  11. Conditions
  12. Results
  13. Related work
  14. Conclusion
  15. Acknowledgement

Figures (2)

  • Figure 1: An overview of training data generation process. Our proposed method cleanses choral singing to solo singing. We can generate training data for the singer diarization from the cleansed data.
  • Figure 2: Trajectories of multiple $F_0$ (upper: raw data, lower: cleansed data), and the reference singer labels. The singer A was a female singing with a higher voice and B was a male singing with a lower voice.