Melodia: Training-Free Music Editing Guided by Attention Probing in Diffusion Models

TL;DR

Melodia addresses the difficulty of text-guided music editing that preserves melody and rhythm by revealing that cross-attention encodes musical attributes while self-attention preserves temporal structure. It introduces a training-free approach that stores source self-attention in an attention repository during partial DDIM inversion and uses Attention-based Structure Retention to guide edits in selected layers, avoiding the need for source prompts. The paper also presents ASB and AMB metrics, along with MelodiaEdit, to quantify the trade-off between textual adherence and structural integrity. Experimental results across multiple datasets and a subjective study demonstrate superior balance and robustness compared to state-of-the-art baselines, with good generalization across models and sampling rates.

Abstract

Text-to-music generation technology is progressing rapidly, creating new opportunities for musical composition and editing. However, existing music editing methods often fail to preserve the source music's temporal structure, including melody and rhythm, when altering particular attributes like instrument, genre, and mood. To address this challenge, this paper conducts an in-depth probing analysis on attention maps within AudioLDM 2, a diffusion-based model commonly used as the backbone for existing music editing methods. We reveal a key finding: cross-attention maps encompass details regarding distinct musical characteristics, and interventions on these maps frequently result in ineffective modifications. In contrast, self-attention maps are essential for preserving the temporal structure of the source music during its conversion into the target music. Building upon this understanding, we present Melodia, a training-free technique that selectively manipulates self-attention maps in particular layers during the denoising process and leverages an attention repository to store source music information, achieving accurate modification of musical characteristics while preserving the original structure without requiring textual descriptions of the source music. Additionally, we propose two novel metrics to better evaluate music editing methods. Both objective and subjective experiments demonstrate that our approach achieves superior results in terms of textual adherence and structural integrity across various datasets. This research enhances comprehension of internal mechanisms within music generation models and provides improved control for music creation.

Figures (6)

• Figure 1: Spectrogram comparison of music editing results between different methods. The comparison reveals that existing methods struggle to preserve the original temporal patterns and rhythmic structures, while Melodia maintains better structural consistency with the source music.
• Figure 2: Results of cross-attention map and self-attention map replacement in different layers of the AudioLDM 2.
• Figure 3: (Left) Overview of Melodia. $\quad$(Right) Detailed explanation of Attention-based Structure Retention (ASR).
• Figure 4: Intuitive Illustration of DDIM Inversion and Reverse Process with Attention Repository based Structure Guidance. The orange and blue paths respectively refer to DDIM Inversion path and reverse path.
• Figure 5: Quantitative Comparison with methods over $T_\text{start}$ range of 300-1000 on MelodiaEdit. The highlighted region is the optimal balance region where shows both text adherence and structural integrity. Our method outperforms other approaches across all $T_\text{start}$ values.