Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models

TL;DR

The paper tackles long-form, well-structured symbolic music generation by introducing a four-level hierarchical music language (Form, Reduced Lead Sheet, Lead Sheet, Accompaniment) and training four cascaded diffusion models, each level conditioned on higher levels and optional external controls. By representing each level as multi-channel image-like piano-rolls and applying top-down conditioning plus autoregressive and cross-attention mechanisms, the approach achieves coherent full-piece generation with recognizable verse-chorus forms and cadences while allowing flexible control via pre-trained latent encoders. Objective metrics (Inter-Phrase Latent Similarity) and subjective listening tests show improvements over strong baselines in both structure and musical quality. The work also analyzes efficiency benefits of cascaded modeling and demonstrates controllability through external latent representations for chords, rhythms, and textures, making the method extensible to broader music-generation tasks. Overall, this hierarchical diffusion framework provides a principled, scalable path toward high-quality, controllable whole-song generation in symbolic music and suggests potential extensions to multi-track and audio domains.

Abstract

Recent deep music generation studies have put much emphasis on long-term generation with structures. However, we are yet to see high-quality, well-structured whole-song generation. In this paper, we make the first attempt to model a full music piece under the realization of compositional hierarchy. With a focus on symbolic representations of pop songs, we define a hierarchical language, in which each level of hierarchy focuses on the semantics and context dependency at a certain music scope. The high-level languages reveal whole-song form, phrase, and cadence, whereas the low-level languages focus on notes, chords, and their local patterns. A cascaded diffusion model is trained to model the hierarchical language, where each level is conditioned on its upper levels. Experiments and analysis show that our model is capable of generating full-piece music with recognizable global verse-chorus structure and cadences, and the music quality is higher than the baselines. Additionally, we show that the proposed model is controllable in a flexible way. By sampling from the interpretable hierarchical languages or adjusting pre-trained external representations, users can control the music flow via various features such as phrase harmonic structures, rhythmic patterns, and accompaniment texture.

Figures (8)

• Figure 1: The diagram of cascaded diffusion models for hierarchical symbolic music generation.
• Figure 2: An example of whole-song generation of 40 measures under a given Form (A$\flat$ major key and "i4A4A4B8b4A4B8o4" phrases). The three staves (from top to bottom) show the generated Reduced Lead Sheet, Lead Sheet, and Accompaniment. Here, rectangles with colored background are used to indicate the appearance of the same motifs in verse and chorus sections. Dashed border rectangles with colored background indicate a variation of motifs. We use red dotted rectangles to show where the generated score show a strong implication of phrase boundary or cadence. The generated chord progressions in Reduced Lead Sheet and Lead Sheet are identical, shown by the chord symbols.
• Figure 3: Subjective evaluation results on music quality and well-structuredness. GT indicates ground truth samples composed by humans.
• Figure 4: An example data representation of our proposed hierarchical music language.
• Figure 5: Examples of generated Reduced Lead Sheet of "A8" phrase in E$\flat$ major. The samples marked with * are controlled by the external condition meaning "unchanging chord progression".