SegTune: Structured and Fine-Grained Control for Song Generation

TL;DR

SegTune tackles the challenge of fine-grained temporal control in song generation by introducing a non-autoregressive Diffusion Transformer that supports both global and segment-level prompts. A dedicated prompt encoder injects segment-local conditioning while a fine-tuned LLM provides sentence-level lyric durations in LRC format, enabling precise lyric-to-music alignment. The authors build a large-scale Mandarin-pop data pipeline with quality filtering, lyrics processing, and segment-aware music captions, plus a unified evaluation framework including MuLan and segment-level metrics. Experimental results show SegTune achieves superior controllability and musical coherence over baselines, with ablations highlighting the importance of prompt encoding strategy and duration prediction accuracy. This work advances programmatic and interactive song generation by enabling structured, segment-aware editing and control in a scalable, end-to-end framework.

Abstract

Recent advancements in song generation have shown promising results in generating songs from lyrics and/or global text prompts. However, most existing systems lack the ability to model the temporally varying attributes of songs, limiting fine-grained control over musical structure and dynamics. In this paper, we propose SegTune, a non-autoregressive framework for structured and controllable song generation. SegTune enables segment-level control by allowing users or large language models to specify local musical descriptions aligned to song sections.The segmental prompts are injected into the model by temporally broadcasting them to corresponding time windows, while global prompts influence the whole song to ensure stylistic coherence. To obtain accurate segment durations and enable precise lyric-to-music alignment, we introduce an LLM-based duration predictor that autoregressively generates sentence-level timestamped lyrics in LRC format. We further construct a large-scale data pipeline for collecting high-quality songs with aligned lyrics and prompts, and propose new evaluation metrics to assess segment-level alignment and vocal attribute consistency. Experimental results show that SegTune achieves superior controllability and musical coherence compared to existing baselines. See https://cai525.github.io/SegTune_demo for demos of our work.

Figures (3)

• Figure 1: The figure illustrates the input format of SegTune, which includes lyrics and textual prompts. The global prompt describes overall musical attributes such as genre, vocal timbre, and global mood tune. Segment prompts specify the attributes of individual segments, such as structure labels, emotion change, or instrumentation. Segment prompts can either be manually provided by the user or automatically generated by the LLM during the prompt engineering stage.
• Figure 2: Overview of the SegTune architecture. The model takes lyrics and textual prompts as input. A LLM-based duration predictor estimates sentence-level durations for the lyrics, while a lyrics encoder embeds the lyrics and performs sentence-level alignment. The prompt encoder encodes both global and segment prompts into 1D vectors. The global prompt is broadcast to all time steps, whereas each segment prompt is broadcast to the frames within its corresponding temporal window, as determined by the duration predictor. All the conditional embeddings are concatenated and fed into a Diffusion Transformer. In the diagram, denotes temporal broadcasting within a segment window, and denotes channel-wise feature concatenation.
• Figure 3: Overview of the data pipeline of SegTune