Streaming Piano Transcription Based on Consistent Onset and Offset Decoding with Sustain Pedal Detection

TL;DR

This work tackles real-time streaming piano transcription by reformulating onset and offset detection as a streaming sequence problem. A CNN-based encoder processes local audio features, while two Transformer-style decoders separately predict onsets and offsets for active notes, with sustain-pedal information guiding offset timing to enforce consistency. On MAESTRO, the method matches or surpasses offline state-of-the-art results while offering substantially lower computational cost and a practical latency of around 380 ms. The results demonstrate the viability of real-time, onset–offset-consistent piano transcription suitable for interactive and latency-sensitive applications. Ablations confirm the importance of pedal-aware duration modeling and the benefit of separating onset and offset decoders.

Abstract

This paper describes a streaming audio-to-MIDI piano transcription approach that aims to sequentially translate a music signal into a sequence of note onset and offset events. The sequence-to-sequence nature of this task may call for the computationally-intensive transformer model for better performance, which has recently been used for offline transcription benchmarks and could be extended for streaming transcription with causal attention mechanisms. We assume that the performance limitation of this naive approach lies in the decoder. Although time-frequency features useful for onset detection are considerably different from those for offset detection, the single decoder is trained to output a mixed sequence of onset and offset events without guarantee of the correspondence between the onset and offset events of the same note. To overcome this limitation, we propose a streaming encoder-decoder model that uses a convolutional encoder aggregating local acoustic features, followed by an autoregressive Transformer decoder detecting a variable number of onset events and another decoder detecting the offset events for the active pitches with validation of the sustain pedal at each time frame. Experiments using the MAESTRO dataset showed that the proposed streaming method performed comparably with or even better than the state-of-the-art offline methods while significantly reducing the computational cost.

Figures (3)

• Figure 1: An overview of the proposed streaming audio-to-MIDI piano transcription method aware of onset-offset correspondence.
• Figure 2: The implementation of the streaming transcription model that uses one encoder for latent feature extraction and two decoders for onset and offset detection.
• Figure :