Piano Transcription by Hierarchical Language Modeling with Pretrained Roll-based Encoders

TL;DR

Automatic Music Transcription (AMT) faces a trade-off between frame-level piano-roll approaches that require thresholding and post-processing and language-model (LM) based methods that contend with long, flattened sequences. The authors propose a hybrid method that fuses pre-trained roll-based encoders with a decoder-only Transformer LM and a hierarchical prediction scheme predicting onset-pitch, velocity, and offset in sequence, reducing the effective sequence length to improve efficiency and accuracy, with a theoretical time complexity of $O(3(T+N)^2D)$ compared to $O((T+3N)^2D)$ for a single model. On Maestro, using two encoders (CRNN and HPPNet) and an LM decoder, the approach yields improvements of about 0.01 and 0.022 in onset-offset-velocity F1 over traditional piano-roll baselines, and shows that encoder choice dominates LM size in performance. The results highlight the importance of encoder design for AMT and suggest that hierarchical, LM-based decoding can be a practical plug-in for roll-based transcription, with future work needed to scale LM-based AMT further.

Abstract

Automatic Music Transcription (AMT), aiming to get musical notes from raw audio, typically uses frame-level systems with piano-roll outputs or language model (LM)-based systems with note-level predictions. However, frame-level systems require manual thresholding, while the LM-based systems struggle with long sequences. In this paper, we propose a hybrid method combining pre-trained roll-based encoders with an LM decoder to leverage the strengths of both methods. Besides, our approach employs a hierarchical prediction strategy, first predicting onset and pitch, then velocity, and finally offset. The hierarchical prediction strategy reduces computational costs by breaking down long sequences into different hierarchies. Evaluated on two benchmark roll-based encoders, our method outperforms traditional piano-roll outputs 0.01 and 0.022 in onset-offset-velocity F1 score, demonstrating its potential as a performance-enhancing plug-in for arbitrary roll-based music transcription encoder.

Figures (3)

• Figure 1: Proposed system architecture. (a) Pre-training stage trains the roll-based encoder with frame-level objectives. (b) LM-training stage connects it to a LM decoder, training with note-level objectives.
• Figure 2: Flattened and hierarchical token sequence. $o_n, p_n, v_n$ and $d_n$ represents a note event with onset time $o_n$, pitch $p_n$, velocity $v_n$, and offset $d_n$. Flattened approach (a) forms one sequence, while the hierarchical approach ($b_1$, $b_2$ and $b_3$) forms three sequences. Two lines on $o$ and $p$ tokens denote that they are fixed from $b_1$ during $b_2$ and $b_3$. Best viewed in color. See Sec. \ref{['sec:flattened']} for details.
• Figure 3: Onset-pitch, velocity and offset loss on the training set (solid lines) and validation set (dotted lines). Best viewed in color.