SteerMusic: Enhanced Musical Consistency for Zero-shot Text-guided and Personalized Music Editing

TL;DR

SteerMusic introduces delta denoising score (DDS) for zero-shot, data-space music editing to preserve original content while aligning edits with target prompts. SteerMusic+ extends this to personalized editing by learning a user-defined concept via a personalized diffusion model and auxiliary losses that maintain source content and enhance concept fidelity. The methods demonstrate superior musical consistency and editing fidelity over state-of-the-art baselines on short and long-form audio, validated by objective metrics and human studies. This work advances zero-shot and personalized music editing by effectively mitigating inversion errors and enabling fine-grained stylistic control.

Abstract

Music editing is an important step in music production, which has broad applications, including game development and film production. Most existing zero-shot text-guided editing methods rely on pretrained diffusion models by involving forward-backward diffusion processes. However, these methods often struggle to preserve the musical content. Additionally, text instructions alone usually fail to accurately describe the desired music. In this paper, we propose two music editing methods that improve the consistency between the original and edited music by leveraging score distillation. The first method, SteerMusic, is a coarse-grained zero-shot editing approach using delta denoising score. The second method, SteerMusic+, enables fine-grained personalized music editing by manipulating a concept token that represents a user-defined musical style. SteerMusic+ allows for the editing of music into user-defined musical styles that cannot be achieved by the text instructions alone. Experimental results show that our methods outperform existing approaches in preserving both music content consistency and editing fidelity. User studies further validate that our methods achieve superior music editing quality.

Figures (9)

• Figure 1: The distortion of the reconstructed melody (CQT1-PCC=0.721) after only 20 DDIM inversion steps.
• Figure 2: SteerMusic: Steering the music style with text-guided music editing or personalized music editing.
• Figure 3: Overview of two music editing pipelines: (a) shows the conventional approach, which performs editing during denoising after an inversion process in the diffusion latent space; (b) shows our solution, which directly edits in data space by optimizing the differentiable function $x = g(\theta)$. The differentiable function is initialized with $x^{\text{src}}$. [$S$] denotes a user-defined concept token, and gray circles represent the optimization trajectory from source to target.
• Figure 4: Overview of the SteerMusic+ pipeline: (a) Personalized diffusion model (PDM) fine-tuned using $\mathcal{D}^{\text{ref}}$ and a user-defined [guitar] concept token. (b) Personalized editing using the PDM $\epsilon_{\phi'}$ from (a). Red dashed lines indicate gradient flows.
• Figure 5: User preference for SteerMusic: percentage of users preferring our method over ZETA and MusicMagus.