DiffRoll: Diffusion-based Generative Music Transcription with Unsupervised Pretraining Capability

TL;DR

This work addresses automatic music transcription (AMT) by reframing it as a conditional diffusion-based generative task, introducing DiffRoll which maps Gaussian noise to a piano-roll posterior $\ ilde{x}_0$ conditioned on mel-spectrograms $c_ ext{mel}$. It employs a DiffWave-inspired architecture and classifier-free guidance (CFG) to enable simultaneous conditional and unconditional generation, training with a denoising objective that minimizes $\ abla L = \|x_ ext{roll}-\hat{x}_0(\cdot)\|^2$. The model demonstrates strong transcription performance, notably outperforming a discriminative baseline by about 19 percentage points in note-wise F1, and shows benefits from unsupervised pretraining on unpaired piano rolls, with results approaching semi-supervised baselines. Beyond transcription, DiffRoll can generate new piano rolls and perform inpainting, showcasing the practical potential of diffusion models for AMT and suggesting future directions such as onset/offset prediction and discrete diffusion for binary data.

Abstract

In this paper we propose a novel generative approach, DiffRoll, to tackle automatic music transcription (AMT). Instead of treating AMT as a discriminative task in which the model is trained to convert spectrograms into piano rolls, we think of it as a conditional generative task where we train our model to generate realistic looking piano rolls from pure Gaussian noise conditioned on spectrograms. This new AMT formulation enables DiffRoll to transcribe, generate and even inpaint music. Due to the classifier-free nature, DiffRoll is also able to be trained on unpaired datasets where only piano rolls are available. Our experiments show that DiffRoll outperforms its discriminative counterpart by 19 percentage points (ppt.) and our ablation studies also indicate that it outperforms similar existing methods by 4.8 ppt. Source code and demonstration are available https://sony.github.io/DiffRoll/.

Figures (5)

• Figure 1: Training phase of DiffRoll. Within each batch, several spectrograms are randomly converted to -1 via the dropout layer.
• Figure 2: Diffusion process of DiffRoll. Forward process converts $x_\text{roll}$ into $x_t$ for $1\le t\le T$ via a Markov process. Reverse process $p_\theta(x_{t-1}|x_t,c_\text{mel})$ is performed iteratively until $x_0$ is reached. A threshold of 0.5 is applied to $x_0$ to obtain the predicted piano roll.
• Figure 3: Result of DiffRoll ablation study with different values of dropout rate $p$. Baseline was discriminative version of DiffRoll, which directly predicts $x_0$.
• Figure 4: Results of CFG ablation study with different values of sampling weight $w$. Discriminative AMT model and U-net model cheuk2021icpr were the baseline models.
• Figure 5: Piano roll inpainting when part of $c_\text{mel}$ is masked by -1 (indicated by red rectangles). DiffRoll generates notes in masked regions and transcribes notes in unmasked regions. More examples are available in the demo.