Quantifying the Corpus Bias Problem in Automatic Music Transcription Systems

TL;DR

The paper investigates corpus bias in Automatic Music Transcription by evaluating state-of-the-art systems trained on classical piano data under controlled musical distribution shifts. It uses Disklavier-based ground-truth data and two targeted test sets to isolate genre and polyphony/dynamics shifts, enabling precise cross-domain evaluation. Results reveal substantial performance degradation under genre shifts and random-note conditions, underscoring persistent corpus bias and generalization gaps. The findings motivate broader, timbre- and genre-diverse training data and rigorous OOD evaluation to ensure AMT systems generalize beyond piano-centric corpora.

Abstract

Automatic Music Transcription (AMT) is the task of recognizing notes in audio recordings of music. The State-of-the-Art (SotA) benchmarks have been dominated by deep learning systems. Due to the scarcity of high quality data, they are usually trained and evaluated exclusively or predominantly on classical piano music. Unfortunately, that hinders our ability to understand how they generalize to other music. Previous works have revealed several aspects of memorization and overfitting in these systems. We identify two primary sources of distribution shift: the music, and the sound. Complementing recent results on the sound axis (i.e. acoustics, timbre), we investigate the musical one (i.e. note combinations, dynamics, genre). We evaluate the performance of several SotA AMT systems on two new experimental test sets which we carefully construct to emulate different levels of musical distribution shift. Our results reveal a stark performance gap, shedding further light on the Corpus Bias problem, and the extent to which it continues to trouble these systems.

Figures (1)

• Figure 1: The Musical OOD performance of SotA systems (Note F1 with Onset, Offset and Velocity).