MEDIC: Zero-shot Music Editing with Disentangled Inversion Control

TL;DR

This work tackles zero-shot music editing with diffusion models by introducing Disentangled Inversion Control (DIC) and Harmonized Attention Control (HAC). DIC splits the diffusion process into three branches (source, harmonic, target) and applies a corrective path to reduce inversion drift, while HAC unifies cross-attention and mutual self-attention through a Harmonic Branch to enable both rigid and non-rigid edits. The authors also provide ZoME-Bench, a comprehensive 1,100-sample benchmark across 10 editing tasks to evaluate edit fidelity and content preservation. Experimental results show MEDIC surpasses state-of-the-art inversion and editing baselines on ZoME-Bench and MusicDelta, with robust performance for long and short edits and thorough ablations confirming the contribution of each component. The work offers a practical, training-free framework for nuanced zero-shot music editing and establishes a standardized benchmark for future research.

Abstract

Text-guided diffusion models revolutionize audio generation by adapting source audio to specific text prompts. However, existing zero-shot audio editing methods such as DDIM inversion accumulate errors across diffusion steps, reducing the effectiveness. Moreover, existing editing methods struggle with conducting complex non-rigid music edits while maintaining content integrity and high fidelity. To address these challenges, we propose MEDIC, a novel zero-shot music editing system based on innovative Disentangled Inversion Control (DIC) technique, which comprises Harmonized Attention Control and Disentangled Inversion. Disentangled Inversion disentangles the diffusion process into triple branches to rectify the deviated path of the source branch caused by DDIM inversion. Harmonized Attention Control unifies the mutual self-attention control and the cross-attention control with an intermediate Harmonic Branch to progressively generate the desired harmonic and melodic information in the target music. We also introduce ZoME-Bench, a comprehensive music editing benchmark with 1,100 samples covering ten distinct editing categories. ZoME-Bench facilitates both zero-shot and instruction-based music editing tasks. Our method outperforms state-of-the-art inversion techniques in editing fidelity and content preservation. The code and benchmark will be released. Audio samples are available at https://medic-edit.github.io/.

Figures (14)

• Figure 1: Spider chart for CLAP score comparisons.
• Figure 2: Spider chart for FAD score comparisons.
• Figure 4: Comparisons between our method and major baselines based on two-branch inversion techniques, including DDIM Inversion song2020denoising and MusicMagus zhang2024musicmagus. (a) Framework of DDIM Inversion, showing configurations with and without classifier-free guidance. (b) Framework of MusicMagus, which incorporates cross-attention control. (c) Framework of our method, featuring Disentangled Inversion Control. (d) An illustration comparing the output of the two-branch techniques with the progressive output of our triple-branch method.
• Figure 5: The Harmonized Attention Control (HAC) framework. HAC unifies cross-attention control and mutual self-attention control with an additional branch named Harmonic Branch to host the desired composition and structural information in the target music.
• Figure 6: Visualizations of the mel-spectrograms of the source audio and the edited audio by different editing methods.