FIGARO: Generating Symbolic Music with Fine-Grained Artistic Control

TL;DR

This work tackles controllable symbolic music generation by introducing self-supervised description-to-sequence learning. FIGARO combines an expert, human-interpretable description with a learned, high-fidelity description from a bar-level VQ-VAE, enabling fine-grained and global control over multi-track, multi-time-signature music via a Transformer auto-encoder. Empirical results show state-of-the-art performance on controllable generation and sample quality, with robust generalization to distributional shifts and favorable subjective evaluations. The approach reduces the barrier to human-guided music creation while maintaining interpretability and flexibility, and the REMI+ extension broadens applicability to diverse musical structures.

Abstract

Generating music with deep neural networks has been an area of active research in recent years. While the quality of generated samples has been steadily increasing, most methods are only able to exert minimal control over the generated sequence, if any. We propose the self-supervised description-to-sequence task, which allows for fine-grained controllable generation on a global level. We do so by extracting high-level features about the target sequence and learning the conditional distribution of sequences given the corresponding high-level description in a sequence-to-sequence modelling setup. We train FIGARO (FIne-grained music Generation via Attention-based, RObust control) by applying description-to-sequence modelling to symbolic music. By combining learned high level features with domain knowledge, which acts as a strong inductive bias, the model achieves state-of-the-art results in controllable symbolic music generation and generalizes well beyond the training distribution.

Figures (6)

• Figure 1: Overview of FIGARO. Dashed lines indicate components that are only used during training of the learned description.
• Figure 2: Example of an expert description. The description contains information about time signature, note density, pitch, velocity and duration as well as which instruments and chords are played throughout each bar.
• Figure 3: Schematic overview of description-to-sequence learning in the context of music.
• Figure 4: Win rates of generated samples against real samples. We compare FIGARO, huang_music_2018, choi_encoding_2020 and wu_musemorphose_2021. Real samples are from the test set.
• Figure 5: Example sequence represented in the REMI+ representation. At the beginning of each bar time signature, tempo and the current chord are noted, after which each note is represented through five subsequent tokens.