Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Knowledge-Driven Approach to Music Segmentation, Music Source Separation and Cinematic Audio Source Separation

Chun-wei Ho, Sabato Marco Siniscalchi, Kai Li, Chin-Hui Lee

TL;DR

A knowledge-driven, model-based approach to segmenting audio into single-category and mixed-category chunks with applications to source separation and Evaluation on simulation data shows that score-guided learning achieves very good music segmentation and separation results.

Abstract

We propose a knowledge-driven, model-based approach to segmenting audio into single-category and mixed-category chunks with applications to source separation. "Knowledge" here denotes information associated with the data, such as music scores. "Model" here refers to tool that can be used for audio segmentation and recognition, such as hidden Markov models. In contrast to conventional learning that often relies on annotated data with given segment categories and their corresponding boundaries to guide the learning process, the proposed framework does not depend on any pre-segmented training data and learns directly from the input audio and its related knowledge sources to build all necessary models autonomously. Evaluation on simulation data shows that score-guided learning achieves very good music segmentation and separation results. Tested on movie track data for cinematic audio source separation also shows that utilizing sound category knowledge achieves better separation results than those obtained with data-driven techniques without using such information.

A Knowledge-Driven Approach to Music Segmentation, Music Source Separation and Cinematic Audio Source Separation

TL;DR

A knowledge-driven, model-based approach to segmenting audio into single-category and mixed-category chunks with applications to source separation and Evaluation on simulation data shows that score-guided learning achieves very good music segmentation and separation results.

Abstract

We propose a knowledge-driven, model-based approach to segmenting audio into single-category and mixed-category chunks with applications to source separation. "Knowledge" here denotes information associated with the data, such as music scores. "Model" here refers to tool that can be used for audio segmentation and recognition, such as hidden Markov models. In contrast to conventional learning that often relies on annotated data with given segment categories and their corresponding boundaries to guide the learning process, the proposed framework does not depend on any pre-segmented training data and learns directly from the input audio and its related knowledge sources to build all necessary models autonomously. Evaluation on simulation data shows that score-guided learning achieves very good music segmentation and separation results. Tested on movie track data for cinematic audio source separation also shows that utilizing sound category knowledge achieves better separation results than those obtained with data-driven techniques without using such information.
Paper Structure (14 sections, 2 figures, 4 tables)

This paper contains 14 sections, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Knowledge-Driven Framework
  4. Model-Based Segment Detection and Selection
  5. Music Source Separation Trained with Segmented Data
  6. Segment-informed CASS
  7. Experimental Setup for Segmentaiton, Separation & CASS
  8. Data Sets and Preprocessing
  9. Music Segmentation and Separation
  10. Cinematic Audio Source Separation
  11. HMM-based Music Alignment Results
  12. Music Source Separation Results
  13. Cinematic AudioSource Separation Results
  14. Summary and Future Work

Figures (2)

  • Figure 1: A knowledge-driven cinematic audio source separation system. The separator can be an RNN, a Transformer, or any separation model. It receives the mixture TF spectrogram concatenated with the projected knowledge and produces the separated spectrograms.
  • Figure 2: Grouping of the 3-D knowledge projection for all possible voice activities in the DNR-nonverbal set for (1) left: speech vs. non-speech; (2) middle: with vs. w/o music; and (3) right: with vs. w/o sound effect.