MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition

TL;DR

MIDI-GPT tackles the gap between powerful generative models and practical, controllable music creation by introducing a transformer-based system with a novel multi-track tokenization approach that supports track- and bar-level infilling and explicit attribute controls. It provides two tokenizations (Multi-Track and Bar-Fill) and adds expressiveness through VELOCITY and DELTA tokens, enabling dynamic and nuanced performances. The model is trained on the GigaMIDI dataset and evaluated for originality, stylistic similarity, and control effectiveness, showing reduced data duplication with longer generations, stylistic alignment with training data, and meaningful control over density, duration, and polyphony. The work demonstrates real-world viability through collaborations with industry partners and tool integrations, and outlines future directions for real-time generation, larger contexts, and broader applicability. Overall, MIDI-GPT advances controllable, cooperative music generation in practical workflows, with demonstrated originality, style preservation, and usable attribute controls.

Abstract

We present and release MIDI-GPT, a generative system based on the Transformer architecture that is designed for computer-assisted music composition workflows. MIDI-GPT supports the infilling of musical material at the track and bar level, and can condition generation on attributes including: instrument type, musical style, note density, polyphony level, and note duration. In order to integrate these features, we employ an alternative representation for musical material, creating a time-ordered sequence of musical events for each track and concatenating several tracks into a single sequence, rather than using a single time-ordered sequence where the musical events corresponding to different tracks are interleaved. We also propose a variation of our representation allowing for expressiveness. We present experimental results that demonstrate that MIDI-GPT is able to consistently avoid duplicating the musical material it was trained on, generate music that is stylistically similar to the training dataset, and that attribute controls allow enforcing various constraints on the generated material. We also outline several real-world applications of MIDI-GPT, including collaborations with industry partners that explore the integration and evaluation of MIDI-GPT into commercial products, as well as several artistic works produced using it.

Figures (6)

• Figure 1: The Multitrack (A) and Bar-Fill (B) tokenizations. The grey $\texttt{<BAR>}$, $\texttt{<TRACK>}$ and $\texttt{<CONTROL>}$ placeholders correspond token subsequences of complete bars, complete tracks, and attribute controls, respectively.
• Figure 2: Adding expressivity to the token sequence
• Figure 3: The percentage of generated excerpts $(g_i)$ with a Hamming distance (resp. Jaccard Index $\mathcal{J}(o_i,g_i)$) between any excerpt $o_i$ from the training dataset and $g_i$ on the range [a,b). A Hamming distance (resp. Jaccard Index) of distance 0 (resp. of 1) indicates two excerpts are identical, while 1 (resp. 0) indicates they are very different.
• Figure 4: The percentage of trials where $\mathcal{S}_{\hat{\mathcal{O}}^{\star}_{25},\hat{\mathcal{G}}^{\star}_{25}}^{\mathcal{C}^{\star}_{50}}(\hat{\mathcal{O}}^{\star}_{25},\mathcal{C}^{\star}_{50}) \leq \mathcal{S}_{\hat{\mathcal{O}}^{\star}_{25},\hat{\mathcal{G}}^{\star}_{25}}^{\mathcal{C}^{\star}_{50}}(\hat{\mathcal{G}}^{\star}_{25},\mathcal{C}^{\star}_{50})$. Hatching indicates that the binomial test was insignificant, indicating that $\hat{\mathcal{O}}^{\star}$ is not more similar to $\mathcal{C}$ than $\hat{\mathcal{G}}^{\star}$.
• Figure 5: The percentage for each absolute difference between requested and actual note density.