SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training

TL;DR

SongGLM addresses two core challenges in lyric-to-melody generation: alignment accuracy and lyric-melody harmony. It introduces a unified symbolic song representation with 2D alignment encoding and a GLM-based multi-task pre-training regime employing hierarchical blank infilling (n-gram, phrase, long span) that leverages lyric-melody relationships to extract harmonized n-grams. A large MelodyNet-derived lyric-melody dataset enables pre-training and fine-tuning, yielding significant improvements over baselines in both objective alignment metrics and perceptual listening tests. The approach demonstrates that direct lyric-melody alignment and multi-scale harmony modeling enhance the quality, diversity, and singability of generated melodies, with potential for multilingual extensions and broader automatic music generation tasks.

Abstract

Lyric-to-melody generation aims to automatically create melodies based on given lyrics, requiring the capture of complex and subtle correlations between them. However, previous works usually suffer from two main challenges: 1) lyric-melody alignment modeling, which is often simplified to one-syllable/word-to-one-note alignment, while others have the problem of low alignment accuracy; 2) lyric-melody harmony modeling, which usually relies heavily on intermediates or strict rules, limiting model's capabilities and generative diversity. In this paper, we propose SongGLM, a lyric-to-melody generation system that leverages 2D alignment encoding and multi-task pre-training based on the General Language Model (GLM) to guarantee the alignment and harmony between lyrics and melodies. Specifically, 1) we introduce a unified symbolic song representation for lyrics and melodies with word-level and phrase-level (2D) alignment encoding to capture the lyric-melody alignment; 2) we design a multi-task pre-training framework with hierarchical blank infilling objectives (n-gram, phrase, and long span), and incorporate lyric-melody relationships into the extraction of harmonized n-grams to ensure the lyric-melody harmony. We also construct a large-scale lyric-melody paired dataset comprising over 200,000 English song pieces for pre-training and fine-tuning. The objective and subjective results indicate that SongGLM can generate melodies from lyrics with significant improvements in both alignment and harmony, outperforming all the previous baseline methods.

Figures (7)

• Figure 1: Illustration of lyric-to-melody generation challenges: alignment modeling and harmony modeling.
• Figure 2: An overview of SongGLM. It includes two stages: harmonized n-gram extraction (detailed in Section \ref{['subsec:harmonized_n-gram_extraction']}) and lyric-to-melody generation (detailed in Section \ref{['subsec:lyric-to-melody_generation']}).
• Figure 3: Examples of melodic peaks in the song.
• Figure 4: Examples of the rhythm skeleton in the song.
• Figure 5: An example of lyric and melodic features, as well as the relationships between them.