Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Facing the Music: Tackling Singing Voice Separation in Cinematic Audio Source Separation

Karn N. Watcharasupat, Chih-Wei Wu, Iroro Orife

TL;DR

This work tackles four-stem cinematic audio source separation (DX, MX-I, MX-V, FX) by extending existing Bandit and Banquet models. It finds that a single shared-decoder Banquet with FiLM-based band-agnostic conditioning outperforms the traditional dedicated-decoder Bandit, improving separation especially for vocal and instrumental components within MX. The approach is evaluated on an adapted DnR v3 dataset built from MUSDB18-HQ and MoisesDB, with ground-truth four-stem supervision; Banquet achieves statistically significant SNR gains and demonstrates effective feature disentanglement through the FiLM mechanism. The results suggest practical benefits for fine-grained cinematic audio control, and the authors provide dataset and model resources for further research under non-commercial licensing.

Abstract

Cinematic audio source separation (CASS), as a standalone problem of extracting individual stems from their mixture, is a fairly new subtask of audio source separation. A typical setup of CASS is a three-stem problem, with the aim of separating the mixture into the dialogue (DX), music (MX), and effects (FX) stems. Given the creative nature of cinematic sound production, however, several edge cases exist; some sound sources do not fit neatly in any of these three stems, necessitating the use of additional auxiliary stems in production. One very common edge case is the singing voice in film audio, which may belong in either the DX or MX or neither, depending heavily on the cinematic context. In this work, we demonstrate a very straightforward extension of the dedicated-decoder Bandit and query-based single-decoder Banquet models to a four-stem problem, treating non-musical dialogue, instrumental music, singing voice, and effects as separate stems. Interestingly, the query-based Banquet model outperformed the dedicated-decoder Bandit model. We hypothesized that this is due to a better feature alignment at the bottleneck as enforced by the band-agnostic FiLM layer. Dataset and model implementation will be made available at https://github.com/kwatcharasupat/source-separation-landing.

Facing the Music: Tackling Singing Voice Separation in Cinematic Audio Source Separation

TL;DR

This work tackles four-stem cinematic audio source separation (DX, MX-I, MX-V, FX) by extending existing Bandit and Banquet models. It finds that a single shared-decoder Banquet with FiLM-based band-agnostic conditioning outperforms the traditional dedicated-decoder Bandit, improving separation especially for vocal and instrumental components within MX. The approach is evaluated on an adapted DnR v3 dataset built from MUSDB18-HQ and MoisesDB, with ground-truth four-stem supervision; Banquet achieves statistically significant SNR gains and demonstrates effective feature disentanglement through the FiLM mechanism. The results suggest practical benefits for fine-grained cinematic audio control, and the authors provide dataset and model resources for further research under non-commercial licensing.

Abstract

Cinematic audio source separation (CASS), as a standalone problem of extracting individual stems from their mixture, is a fairly new subtask of audio source separation. A typical setup of CASS is a three-stem problem, with the aim of separating the mixture into the dialogue (DX), music (MX), and effects (FX) stems. Given the creative nature of cinematic sound production, however, several edge cases exist; some sound sources do not fit neatly in any of these three stems, necessitating the use of additional auxiliary stems in production. One very common edge case is the singing voice in film audio, which may belong in either the DX or MX or neither, depending heavily on the cinematic context. In this work, we demonstrate a very straightforward extension of the dedicated-decoder Bandit and query-based single-decoder Banquet models to a four-stem problem, treating non-musical dialogue, instrumental music, singing voice, and effects as separate stems. Interestingly, the query-based Banquet model outperformed the dedicated-decoder Bandit model. We hypothesized that this is due to a better feature alignment at the bottleneck as enforced by the band-agnostic FiLM layer. Dataset and model implementation will be made available at https://github.com/kwatcharasupat/source-separation-landing.
Paper Structure (7 sections, 3 figures, 1 table)

This paper contains 7 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Data
  3. System
  4. Results
  5. Conclusion
  6. Acknowledgments
  7. Ethics Statement

Figures (3)

  • Figure 1: Model architecture for (left) Bandit and (right) Banquet. Bandit has a dedicated decoder for each stem. Banquet uses only one shared decoder.
  • Figure 2: Normalized clustermap of Banquet's $\bm{\gamma}_i$ for each stem in the split MX setup. Lighter color indicates larger normalized value.
  • Figure :