Filling MIDI Velocity using U-Net Image Colorizer

TL;DR

This work addresses MIDI velocity prediction to enhance expressiveness while preserving timing by reframing MIDI as an image colorization problem. It introduces a U‑Net with window attention to predict velocity rolls from three image-like MIDI matrices (onsets, frames, and velocity) and employs a novel loss that combines binary cross‑entropy with cosine similarity, augmented by onset masking and a velocity‑based weighting. The approach is evaluated on MAESTRO v3 and the Saarland SMD dataset, showing superior objective metrics and listening‑test scores compared with ConvAE, Seq2Seq, and a flat baseline, with cross‑dataset results indicating reasonable generalization within piano data. The study highlights $SD_{velo}$ as a meaningful proxy for expressiveness and suggests that image‑based representations may offer advantages over sequential models for MIDI velocity prediction, while noting limitations to piano data and the need for broader instrument generalization in future work.

Abstract

Modern music producers commonly use MIDI (Musical Instrument Digital Interface) to store their musical compositions. However, MIDI files created with digital software may lack the expressive characteristics of human performances, essentially leaving the velocity parameter - a control for note loudness - undefined, which defaults to a flat value. The task of filling MIDI velocity is termed MIDI velocity prediction, which uses regression models to enhance music expressiveness by adjusting only this parameter. In this paper, we introduce the U-Net, a widely adopted architecture in image colorization, to this task. By conceptualizing MIDI data as images, we adopt window attention and develop a custom loss function to address the sparsity of MIDI-converted images. Current dataset availability restricts our experiments to piano data. Evaluated on the MAESTRO v3 and SMD datasets, our proposed method for filling MIDI velocity outperforms previous approaches in both quantitative metrics and qualitative listening tests.

Figures (4)

• Figure 1: Comparison between MIDI notes with human performed velocity versus Music Software default velocity (64 if user not specified). The standard deviation of velocity ($SD_{\text{velo}}$) represents the dispersion of velocities around their mean across the pitches.
• Figure 2: Proposed U-Net architecture. Model input ($F$ roll) comprises 88 pitch bins and 96 time frames. Attn block denotes the windowed scaled dot-product attention. The final velocity roll ($V$ roll) is generated during post-processing by extracting velocity at note positions, and then assigning each note the velocity at its onset.
• Figure 3: MIDI data distributions of the MAESTRO (blue) and SMD (orange) datasets, with density maps highlighting the similarity in their MIDI feature correlations.
• Figure 4: Comparison of human-performed velocity with ConvAE and U-Net predictions. The U-Net result is more human-like than ConvAE, but for Chopin Op. 10 No. 4, accuracy metrics fail to reflect this.