BERT-like Pre-training for Symbolic Piano Music Classification Tasks

TL;DR

The paper addresses symbolic piano music classification under data-scarce conditions by adopting BERT-style masked language modelling for MIDI. It introduces two Transformer models pre-trained on public MIDI datasets (scores and performances) and fine-tunes them on four downstream tasks spanning note-level and sequence-level predictions. The study demonstrates that pre-training with MLM improves accuracy over RNN baselines across most tasks, with CP token representations generally outperforming REMI due to shorter sequences and richer token grouping. While melody-focused tasks benefit the most, style and emotion classification show competitive gains, though emotion performance can be surpass by MusicBERT with larger pre-training data. The work provides a public benchmark, releases code, and outlines future extensions to broader MIDI scenarios and tasks.

Abstract

This article presents a benchmark study of symbolic piano music classification using the masked language modelling approach of the Bidirectional Encoder Representations from Transformers (BERT). Specifically, we consider two types of MIDI data: MIDI scores, which are musical scores rendered directly into MIDI with no dynamics and precisely aligned with the metrical grid notated by its composer and MIDI performances, which are MIDI encodings of human performances of musical scoresheets. With five public-domain datasets of single-track piano MIDI files, we pre-train two 12-layer Transformer models using the BERT approach, one for MIDI scores and the other for MIDI performances, and fine-tune them for four downstream classification tasks. These include two note-level classification tasks (melody extraction and velocity prediction) and two sequence-level classification tasks (style classification and emotion classification). Our evaluation shows that the BERT approach leads to higher classification accuracy than recurrent neural network (RNN)-based baselines.

Figures (7)

• Figure 1: An example of a piece of score encoded using the proposed simplified version of the (a) REMI and (b) CP representations, using seven types of tokens, Bar, Sub-bar, Pitch, Velocity, Duration, Tempo and Pad (not shown here), for piano-only MIDI performance. The text inside parentheses indicates the value each token takes. While each time step corresponds to a single token in REMI, each time step would correspond to a super token that assembles four tokens in total in CP. Without such a token grouping, the sequence length (in terms of the number of time steps) of REMI is longer than that of CP (in this example, 16 versus 4). Please note that the actual scores employed in our work are not as simple as this example as they are polyphonic.
• Figure 2: Illustration of the (a) pre-training procedure of our model for a CP sequence, where the model learns to predict (reconstruct) randomly-picked super tokens masked in the input sequence (each consisting of four tokens, as the example one shown in the middle with time step $t$); and (b), (c) the fine-tuning procedure for note-level and sequence-level classification. Apart from the last few output layers, both pre-training and fine-tuning use the same architecture.
• Figure 3: Confusion tables (in %) for two models for three-class melody classification, calculated on the test split of POP909$_\text{4/4}$. Each row represents the percentage of notes in an actual class while each column represents a predicted class. Notation---"M1": vocal melody, "M2": instrumental melody, "A": accompaniment.
• Figure 4: The melody/non-melody classification result for "POP909-596.mid" by (b) "skyline" chia01skyline, (c) Simonetta et al.'s CNN simonettaCNW19 and (d) our model (performance) + CP. Directing attention to the red circled region within the pianoroll representation, it is evident that the CNN baseline faces challenges in effectively distinguishing between melody and accompaniment, particularly when note pitches reside within the C4 to C5 range during the initial phase. This is especially pronounced in low-pitch scenarios, where the CNN baseline struggles with accurate classification. In contrast, our model exhibits a notably improved predictive accuracy, closely aligning with the ground truth representation. To further supplement the information , the generated melody audio files and pianoroll figures are available in our repository.
• Figure 5: Confusion tables (in %) for velocity prediction, calculated on the test split of POP909$_\text{4/4}$. Each row represents the percentage of notes in an actual class while each column represents a predicted class.