Prompt-guided Precise Audio Editing with Diffusion Models

TL;DR

This work introduces PPAE, a training-free framework for precise audio editing with diffusion models that manipulates cross-attention maps under edited textual prompts. It integrates inversion, a latent diffusion backbone, and a hierarchical local-global editing controller with a Fuser and guidance bootstrapping to enable targeted edits while preserving surrounding structure. The method supports Audio Replace, Audio Refine, Audio Reweight, and even Audio Refusion, demonstrating quantitative gains across FD, FAD, KL, and CLAP metrics, along with favorable subjective assessments. The approach is model-agnostic and offers a flexible, fine-grained editing interface for audio content, with practical implications for media production and notable considerations around ethical use and safety.

Abstract

Audio editing involves the arbitrary manipulation of audio content through precise control. Although text-guided diffusion models have made significant advancements in text-to-audio generation, they still face challenges in finding a flexible and precise way to modify target events within an audio track. We present a novel approach, referred to as PPAE, which serves as a general module for diffusion models and enables precise audio editing. The editing is based on the input textual prompt only and is entirely training-free. We exploit the cross-attention maps of diffusion models to facilitate accurate local editing and employ a hierarchical local-global pipeline to ensure a smoother editing process. Experimental results highlight the effectiveness of our method in various editing tasks.

Figures (26)

• Figure 1: Precise audio editing. Such editing requires modifying the target events while preserving the unrelated events and keeping the overall structure unchanged.
• Figure 2: The overview of the proposed method. Given the edit instruction, the source audio will first be inverted into the given diffusion model's domain, and then edited on the attention-map level under the guidance of our editing controller. The controller accomplishes precise editing by utilizing hierarchical guidance throughout the diffusion process. The whole editing pipeline is training-free and is adaptable to common diffusion models.
• Figure 3: Case Study (Audio Replace)
• Figure 4: Case Study (method compared with Audit)
• Figure 5: Case Study (Audio Refine)