Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation

Karn N. Watcharasupat, Chih-Wei Wu, Yiwei Ding, Iroro Orife, Aaron J. Hipple, Phillip A. Williams, Scott Kramer, Alexander Lerch, William Wolcott

TL;DR

BandIt extends the Bandsplit RNN to flexible, overlapping, psychoacoustically motivated frequency partitions and introduces a common-encoder design for multi-stem cinematic audio source separation. It combines a bandwise feature embedding, residual GRU time–frequency modeling, and an overlapping mask recombination strategy with a novel L1SNR loss, achieving state-of-the-art results on the Divide and Remaster dataset while reducing parameters. The approach demonstrates robust generalization to unseen stems and data, and offers favorable computational trade-offs relative to strong baselines. Collectively, these contributions advance practical CASS by enabling scalable, interpretable, and efficient multi-stem separation with flexible band definitions and detachable decoders.

Abstract

Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.

A Generalized Bandsplit Neural Network for Cinematic Audio Source Separation

TL;DR

BandIt extends the Bandsplit RNN to flexible, overlapping, psychoacoustically motivated frequency partitions and introduces a common-encoder design for multi-stem cinematic audio source separation. It combines a bandwise feature embedding, residual GRU time–frequency modeling, and an overlapping mask recombination strategy with a novel L1SNR loss, achieving state-of-the-art results on the Divide and Remaster dataset while reducing parameters. The approach demonstrates robust generalization to unseen stems and data, and offers favorable computational trade-offs relative to strong baselines. Collectively, these contributions advance practical CASS by enabling scalable, interpretable, and efficient multi-stem separation with flexible band definitions and detachable decoders.

Abstract

Cinematic audio source separation is a relatively new subtask of audio source separation, with the aim of extracting the dialogue, music, and effects stems from their mixture. In this work, we developed a model generalizing the Bandsplit RNN for any complete or overcomplete partitions of the frequency axis. Psychoacoustically motivated frequency scales were used to inform the band definitions which are now defined with redundancy for more reliable feature extraction. A loss function motivated by the signal-to-noise ratio and the sparsity-promoting property of the 1-norm was proposed. We additionally exploit the information-sharing property of a common-encoder setup to reduce computational complexity during both training and inference, improve separation performance for hard-to-generalize classes of sounds, and allow flexibility during inference time with detachable decoders. Our best model sets the state of the art on the Divide and Remaster dataset with performance above the ideal ratio mask for the dialogue stem.
Paper Structure (26 sections, 19 equations, 3 figures, 2 tables)

This paper contains 26 sections, 19 equations, 3 figures, 2 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. PROPOSED METHOD
  4. Common Encoder
  5. Bandsplit Module
  6. Mel Scale
  7. Bark scale
  8. Equivalent Rectangular Bandwidth Scale
  9. 12-TET Western Musical Scale
  10. Bandwise Feature Embedding
  11. Time Frequency Modeling
  12. Overlapping Mask Estimation and Recombination
  13. Loss function
  14. EXPERIMENTAL SETUP
  15. Dataset
  16. ...and 11 more sections

Figures (3)

  • Figure 1: Overview of the proposed model architecture, Bandit.
  • Figure 2: Frequency ranges of each band, by band type, for a 64-band setup with a sampling rate of 44.1kHz and an FFT size of 2048samples.
  • Figure 3: A simplified illustration of overlapping mask recombination.