Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

Ye Bai, Chenxing Li, Hao Li, Yuanyuan Zhao, Xiaorui Wang

TL;DR

The paper tackles recognizing speech and singing simultaneously in overlapping audio by introducing JRSV, which jointly performs multi-task audio source separation (MTASS) and ASR. The MTASS module separates speech and singing while suppressing background music, and a CTC/attention-based ASR decodes each track; online distillation and a two-stage training curriculum stabilize learning. A new DTSSV benchmark is built from AISHELL-1, OpenSinger, and MusDB18, and experiments show large relative CER reductions (up to 41% for speech and 57% for singing) over cascade baselines. The approach demonstrates that separating into track-specific streams before recognition yields substantial gains, with the frozen-MTASS variant and distillation delivering the strongest performance and a practical path for robust two-track recognition in real-world media content.

Abstract

In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.

Jointly Recognizing Speech and Singing Voices Based on Multi-Task Audio Source Separation

TL;DR

The paper tackles recognizing speech and singing simultaneously in overlapping audio by introducing JRSV, which jointly performs multi-task audio source separation (MTASS) and ASR. The MTASS module separates speech and singing while suppressing background music, and a CTC/attention-based ASR decodes each track; online distillation and a two-stage training curriculum stabilize learning. A new DTSSV benchmark is built from AISHELL-1, OpenSinger, and MusDB18, and experiments show large relative CER reductions (up to 41% for speech and 57% for singing) over cascade baselines. The approach demonstrates that separating into track-specific streams before recognition yields substantial gains, with the frozen-MTASS variant and distillation delivering the strongest performance and a practical path for robust two-track recognition in real-world media content.

Abstract

In short video and live broadcasts, speech, singing voice, and background music often overlap and obscure each other. This complexity creates difficulties in structuring and recognizing the audio content, which may impair subsequent ASR and music understanding applications. This paper proposes a multi-task audio source separation (MTASS) based ASR model called JRSV, which Jointly Recognizes Speech and singing Voices. Specifically, the MTASS module separates the mixed audio into distinct speech and singing voice tracks while removing background music. The CTC/attention hybrid recognition module recognizes both tracks. Online distillation is proposed to improve the robustness of recognition further. To evaluate the proposed methods, a benchmark dataset is constructed and released. Experimental results demonstrate that JRSV can significantly improve recognition accuracy on each track of the mixed audio.
Paper Structure (18 sections, 4 equations, 1 figure, 4 tables)

This paper contains 18 sections, 4 equations, 1 figure, 4 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Proposed methods
  4. The MTASS module
  5. The ASR module
  6. Training
  7. MTASS losses
  8. ASR losses
  9. Two-stage training
  10. Inference
  11. DTSSV dataset
  12. Evaluation
  13. Experiments and analysis
  14. Setup
  15. Preliminary experiment: directly recognizing the mixture
  16. ...and 3 more sections

Figures (1)

  • Figure 1: An overview of JRSV system. $\widetilde{\mathbf{s}}$ denotes the mixed audio wave. $\widetilde{\mathbf{X}}$ denotes the mixed spectral magnitude. $\hat{\mathbf{X}}_{\text{speech}}$ and $\hat{\mathbf{X}}_{\text{singing}}$ denote the separated spectral magnitudes of speech and the singing voice respectively. $y_{\text{speech}}$ and $y_{\text{singing}}$ denote the text sequence of speech and singing voice.