SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor

TL;DR

SongEditor extends a zero-shot song generation framework by integrating editing capabilities into a language-model–based system. It combines a semantic tokenization, a decoder-only LM, and a diffusion-based generator to support segment-wise and track-wise edits, including generation from scratch and multi-source track completion. Key innovations include context-free segment editing with rearrangement, force-smoothing training, score-based candidate selection, and a gated multi-source encoder for vocal/accompaniment conditioning. The approach demonstrates strong objective and subjective performance on end-to-end editing tasks and enables long-form, multi-singer storytelling modes, with practical implications for controllable, efficient song production. The work also discusses limitations and ethical considerations, such as track decoupling and copyright concerns, and provides extensive experimental details and appendices for reproducibility.

Abstract

The emergence of novel generative modeling paradigms, particularly audio language models, has significantly advanced the field of song generation. Although state-of-the-art models are capable of synthesizing both vocals and accompaniment tracks up to several minutes long concurrently, research about partial adjustments or editing of existing songs is still underexplored, which allows for more flexible and effective production. In this paper, we present SongEditor, the first song editing paradigm that introduces the editing capabilities into language-modeling song generation approaches, facilitating both segment-wise and track-wise modifications. SongEditor offers the flexibility to adjust lyrics, vocals, and accompaniments, as well as synthesizing songs from scratch. The core components of SongEditor include a music tokenizer, an autoregressive language model, and a diffusion generator, enabling generating an entire section, masked lyrics, or even separated vocals and background music. Extensive experiments demonstrate that the proposed SongEditor achieves exceptional performance in end-to-end song editing, as evidenced by both objective and subjective metrics. Audio samples are available in https://cypress-yang.github.io/SongEditor_demo/.

Figures (5)

• Figure 1: SongEditor supports various song generation and editing tasks, involving creating complete songs from scratch and Infilling Editing. Accomp-to-song and vocal-to-song mean generating full songs based on partial input conditions such as accompaniments or vocals, respectively.
• Figure 2: The architecture of the proposed SongEditor framework. We train the DiT and RVQ jointly first and then the semantic language model. The multi-source encoder is exclusively used for track-wise editing.
• Figure 3: Mel spectrograms of transitions. The center of the red box is the transition point. The left half is generated while the right half is restored from ground truth.
• Figure 4: An example of track-wise editing. The Mel spectrogram above corresponds to the separated accompaniment, while the below corresponds to the vocal. Since the spectrogram of accompaniment is more complex and difficult to identify, its chroma change trend is plotted (red line).
• Figure 5: Text token representation for the structure-guided lyrics.