Separate This, and All of these Things Around It: Music Source Separation via Hyperellipsoidal Queries

TL;DR

Musical source separation has been constrained by a fixed-stem paradigm, limiting target flexibility. The authors propose a query-by-region framework using hyperellipsoidal queries in a PCA-reduced PaSST embedding space, with FiLM conditioning at a complex-valued TF masking network to extract arbitrary target sources. The method combines reconstruction and level-matching losses, enabling stable training and high-fidelity outputs, and demonstrates state-of-the-art SNR and retrieval metrics on MoisesDB, including strong performance on long-tail instruments. This approach offers practical flexibility for creative audio tasks by enabling region-based, multi-source queries beyond predefined stems, with potential extensions to negative queries and automatic region formation.

Abstract

Music source separation is an audio-to-audio retrieval task of extracting one or more constituent components, or composites thereof, from a musical audio mixture. Each of these constituent components is often referred to as a "stem" in literature. Historically, music source separation has been dominated by a stem-based paradigm, leading to most state-of-the-art systems being either a collection of single-stem extraction models, or a tightly coupled system with a fixed, difficult-to-modify, set of supported stems. Combined with the limited data availability, advances in music source separation have thus been mostly limited to the "VDBO" set of stems: \textit{vocals}, \textit{drum}, \textit{bass}, and the catch-all \textit{others}. Recent work in music source separation has begun to challenge the fixed-stem paradigm, moving towards models able to extract any musical sound as long as this target type of sound could be specified to the model as an additional query input. We generalize this idea to a \textit{query-by-region} source separation system, specifying the target based on the query regardless of how many sound sources or which sound classes are contained within it. To do so, we propose the use of hyperellipsoidal regions as queries to allow for an intuitive yet easily parametrizable approach to specifying both the target (location) as well as its spread. Evaluation of the proposed system on the MoisesDB dataset demonstrated state-of-the-art performance of the proposed system both in terms of signal-to-noise ratios and retrieval metrics.

Figures (11)

• Figure 1: Overview of the Proposed System
• Figure 2: A two-dimensional cross-section of a hyperellipsoid in $\mathbb{R}^K$, $K\ge 3$, via a hyperplane $\{\mathbf{w} \in \mathbb{R}^{K} \colon \mathbf{p}_k^\mathsf{T}(\mathbf{w}-\mathbf{c}) = 0,\ \forall k\ne i,j \}$.
• Figure 3: Simplified two-dimensional representation of some possible training queries from the same training data point, meaning that all source embeddings are the same across the three cases. For each case, the valid query with the same input mixture and the target is any ellipse interpolating between the inner (blue) and outer (red) dotted lines. Left: The target is a composite of four sources. The input mixture is a composite of all available sources. Center: The target is a composite of five sources. Not all available sources are used in the mixture. Right: The target is a single source. The input mixture is a composite of all available sources.
• Figure 4: Single-source queries: SNR (dB) distribution by target "stem" over query scale factors, $\alpha$, and the clip-wise best factor.
• Figure 5: Single-source queries: Plot of the median SNRs (dB) and against median RMS errors (dB) of the proposed method and Banquet (q:all, TE+DA variant). Note again that our method was evaluated clip-wise while Banquet was evaluated over the full track.