Conditional Diffusion as Latent Constraints for Controllable Symbolic Music Generation
Matteo Pettenó, Alessandro Ilic Mezza, Alberto Bernardini
TL;DR
This work introduces latent diffusion as latent constraints (LC) to enable post-hoc, fader-like control over an unconditional symbolic-music generator. By treating diffusion models as plug-and-play priors, the LC-Diff library trains small conditional denoisers that steer the frozen backbone’s latent codes toward target attributes (Contour, Note Density, Pitch Range, Rhythm Complexity) without retraining the backbone. Across a four-bar monophonic dataset derived from the Lakh MIDI collection, LC-Diff outperforms attribute-regularized VAEs and LC baselines on both controllability (higher correlations between desired and generated attributes) and fidelity (lower Fréchet Music Distance). The approach demonstrates that diffusion-based latent constraints offer flexible, scalable control over multiple musical attributes, with practical potential for interactive, attribute-guided music composition. Future work will broaden attribute coverage, explore real-time interfaces, and extend LC to other generative paradigms.
Abstract
Recent advances in latent diffusion models have demonstrated state-of-the-art performance in high-dimensional time-series data synthesis while providing flexible control through conditioning and guidance. However, existing methodologies primarily rely on musical context or natural language as the main modality of interacting with the generative process, which may not be ideal for expert users who seek precise fader-like control over specific musical attributes. In this work, we explore the application of denoising diffusion processes as plug-and-play latent constraints for unconditional symbolic music generation models. We focus on a framework that leverages a library of small conditional diffusion models operating as implicit probabilistic priors on the latents of a frozen unconditional backbone. While previous studies have explored domain-specific use cases, this work, to the best of our knowledge, is the first to demonstrate the versatility of such an approach across a diverse array of musical attributes, such as note density, pitch range, contour, and rhythm complexity. Our experiments show that diffusion-driven constraints outperform traditional attribute regularization and other latent constraints architectures, achieving significantly stronger correlations between target and generated attributes while maintaining high perceptual quality and diversity.