The Inverse Drum Machine: Source Separation Through Joint Transcription and Analysis-by-Synthesis

TL;DR

This work tackles drum source separation from mixtures without relying on isolated stems by leveraging transcription annotations. It introduces IDM, an analysis-by-synthesis multitask model that jointly learns automatic drum transcription and one-shot drum sample synthesis conditioned on a mixture embedding to reconstruct individual drum tracks. Trained with a reconstruction objective plus transcription and embedding losses, IDM achieves separation performance comparable to state-of-the-art supervised methods on StemGMD while using far fewer parameters. The approach offers flexible synthesis- and masking-based separation, with potential applications in music production, remixing, and educational materials, especially in data regimes lacking clean multitrack drum stems.

Abstract

We present the Inverse Drum Machine, a novel approach to Drum Source Separation that leverages an analysis-by-synthesis framework combined with deep learning. Unlike recent supervised methods that require isolated stem recordings for training, our approach is trained on drum mixtures with only transcription annotations. IDM integrates Automatic Drum Transcription and One-shot Drum Sample Synthesis, jointly optimizing these tasks in an end-to-end manner. By convolving synthesized one-shot samples with estimated onsets, akin to a drum machine, we reconstruct the individual drum stems and train a Deep Neural Network on the reconstruction of the mixture. Experiments on the StemGMD dataset demonstrate that IDM achieves separation quality comparable to state-of-the-art supervised methods that require isolated stems data.

Figures (7)

• Figure 1: The proposed separation model processes the audio input through several stages. First, a Feature Extraction module extracts learned frame-level features. These features are transformed by a Synthesis Conditioning module into relevant synthesis parameters: transcription onsets, velocities, a mixture embedding, and individual track gains. A decoder module synthesizes one-shot samples for each drum instrument conditioned on the mixture embedding and reconstructs individual drum tracks by sequencing these one-shots with the obtained transcription. The framework encompasses three interconnected tasks: ADT, OSS, and DSS. DSS is obtained either from the decoder output (synthesis estimate) or after time-frequency masking (masked estimate).
• Figure 2: Detailed diagram of the analysis-by-synthesis pipeline, containing the Feature Extraction (blue), Synthesis Conditioning (yellow), and Decoder (purple) modules. Blocks in blue indicate trainable components, blocks in orange indicate differentiable operations, and blocks in pink indicate non-differentiable operations. Loss functions are represented as green components, and grey components represent external information used during training
• Figure 3: One-shot synth model architecture. White noise is fed to a TCN conditioned via FiLM on a conditioning vector $\mathbf{c}$, which has disentangled instrument class/timbre dimensions. Causal zero padding is applied. The output is shaped by a parametrized exponential envelope estimated from $\mathbf{c}$ and normalized to $[-1, 1]$ amplitude range. $\mathbf{c}$ controls the drum kit (mixture embedding) and drum instrument (class one-hot) synthesized by the TCN.
• Figure 4: Log-magnitude spectrograms of synthesized, one-second-long one-shot synthesized (top) and real (bottom) samples for three instruments. The real samples are taken from the StemGMD single hits partition with the second-highest velocity. Y axis is scaled in log frequency for better visibility, and warmer colors represent higher intensity.
• Figure 5: Performance of the transcription module.