Count The Notes: Histogram-Based Supervision for Automatic Music Transcription

TL;DR

CountEM addresses automatic music transcription under weak supervision by replacing frame-level labels with note event histograms. It employs an EM loop in which the E-step refines onset labels via peak-picking on the predicted posterior $Z$ to match a target histogram $h$, and the M-step updates the network with a weighted binary cross-entropy loss. This histogram-based supervision eliminates the need for DTW-based alignment, improving robustness and efficiency while achieving competitive results across piano (MAESTRO), guitar (GuitarSet, GAPS), and multi-instrument (MusicNet) datasets. The approach is notably robust to histogram noise and demonstrates strong cross-dataset generalization, enabling scalable transcription in diverse musical contexts.

Abstract

Automatic Music Transcription (AMT) converts audio recordings into symbolic musical representations. Training deep neural networks (DNNs) for AMT typically requires strongly aligned training pairs with precise frame-level annotations. Since creating such datasets is costly and impractical for many musical contexts, weakly aligned approaches using segment-level annotations have gained traction. However, existing methods often rely on Dynamic Time Warping (DTW) or soft alignment loss functions, both of which still require local semantic correspondences, making them error-prone and computationally expensive. In this article, we introduce CountEM, a novel AMT framework that eliminates the need for explicit local alignment by leveraging note event histograms as supervision, enabling lighter computations and greater flexibility. Using an Expectation-Maximization (EM) approach, CountEM iteratively refines predictions based solely on note occurrence counts, significantly reducing annotation efforts while maintaining high transcription accuracy. Experiments on piano, guitar, and multi-instrument datasets demonstrate that CountEM matches or surpasses existing weakly supervised methods, improving AMT's robustness, scalability, and efficiency. Our project page is available at https://yoni-yaffe.github.io/count-the-notes.

Figures (2)

• Figure 1: Estimating aligned labels from histograms by peak-picking. For each note in the histogram, the $K$ most likely timings are selected according to the current predicted posteriorgram. Since misaligned labels reduce to the same histogram (top), possible timing inaccuracies common in weakly-aligned labels can be overcome.
• Figure :