Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model

Ye Tao, Wen Wu, Chao Zhang, Mengyue Wu, Shuai Wang, Xuenan Xu

TL;DR

MMEdit tackles the challenge of text-guided audio editing by unifying instruction understanding and audio manipulation through an Audio Language Model. It extends editing tasks to six operations, builds a scalable synthetic data pipeline with event-level annotations, and leverages a Qwen2-Audio encoder with an MMDiT-based latent diffusion backbone to achieve precise, context-aware edits. The approach achieves superior instruction following and localizes edits more accurately while preserving non-edited content, outperforming existing baselines on both objective metrics and real-recording evaluations. By open-sourcing its benchmark and data pipeline, the work lays a reproducible foundation for broader research in audio editing with multimodal guidance.

Abstract

Text-guided audio editing aims to modify specific acoustic events while strictly preserving non-target content. Despite recent progress, existing approaches remain fundamentally limited. Training-free methods often suffer from signal degradation caused by diffusion inversion, while training-based methods, although achieving higher generation quality, are severely constrained by the scarcity of high-quality paired data and task formulations that cover only a narrow subset of editing operations. In addition, standard architectures typically decouple text and audio processing, limiting the ability to align instructions with specific acoustic contexts. To address these challenges, we propose MMEdit, an audio-language-model-driven framework for unified audio editing. We systematically extend task definitions to cover a comprehensive range of editing operations, including addition, replacement, removal, reordering, and attribute modification. Furthermore, we design a scalable data synthesis pipeline to construct large-scale paired datasets with fine-grained event-level annotations. To capture complex editing semantics, we integrate a Qwen2-Audio encoder with an MMDiT-based generator, enabling precise cross-modal alignment and localized editing. Experimental results demonstrate that our method achieves superior editing localization accuracy, robust instruction following, and high fidelity in non-edited regions.

MMEDIT: A Unified Framework for Multi-Type Audio Editing via Audio Language Model

TL;DR

MMEdit tackles the challenge of text-guided audio editing by unifying instruction understanding and audio manipulation through an Audio Language Model. It extends editing tasks to six operations, builds a scalable synthetic data pipeline with event-level annotations, and leverages a Qwen2-Audio encoder with an MMDiT-based latent diffusion backbone to achieve precise, context-aware edits. The approach achieves superior instruction following and localizes edits more accurately while preserving non-edited content, outperforming existing baselines on both objective metrics and real-recording evaluations. By open-sourcing its benchmark and data pipeline, the work lays a reproducible foundation for broader research in audio editing with multimodal guidance.

Abstract

Text-guided audio editing aims to modify specific acoustic events while strictly preserving non-target content. Despite recent progress, existing approaches remain fundamentally limited. Training-free methods often suffer from signal degradation caused by diffusion inversion, while training-based methods, although achieving higher generation quality, are severely constrained by the scarcity of high-quality paired data and task formulations that cover only a narrow subset of editing operations. In addition, standard architectures typically decouple text and audio processing, limiting the ability to align instructions with specific acoustic contexts. To address these challenges, we propose MMEdit, an audio-language-model-driven framework for unified audio editing. We systematically extend task definitions to cover a comprehensive range of editing operations, including addition, replacement, removal, reordering, and attribute modification. Furthermore, we design a scalable data synthesis pipeline to construct large-scale paired datasets with fine-grained event-level annotations. To capture complex editing semantics, we integrate a Qwen2-Audio encoder with an MMDiT-based generator, enabling precise cross-modal alignment and localized editing. Experimental results demonstrate that our method achieves superior editing localization accuracy, robust instruction following, and high fidelity in non-edited regions.
Paper Structure (27 sections, 7 equations, 3 figures, 3 tables)

This paper contains 27 sections, 7 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Data pipeline
  3. Audio Segment Curation
  4. Editing Task Definition and Composition
  5. Addition
  6. Removal
  7. Replacement
  8. Reordering
  9. Loudness Adjustment
  10. Speed Modification
  11. MMEDIT
  12. Multimodal Understanding
  13. The MMDiT Backbone Architecture
  14. Global Conditioning via Adaptive Modulation
  15. MMDiT Blocks Design
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Construction pipeline and examples. (1) Extract clean event-level segments from AudioCaps using TAG model and CLAP filtering; (2) Simulate foreground–background compositions and record metadata; (3) Generate template-based instructions for each editing task.
  • Figure 2: Overall architecture of the proposed MMEdit framework. The model employs the powerful Qwen2-Audio for joint understanding of audio and instruction, generating the high-level multi-modal representation $\mathbf{H}$. Then the MMDiT backbone predicts the audio latent $\mathbf{z}_0$ from a Gaussian latent conditioned on $\mathbf{H}$ and the latent of the source audio $\mathbf{z}_\mathrm{in}$. Finally, the VAE decoder converts $\mathbf{z}_\mathrm{0}$ to the edited audio.
  • Figure 3: Effect of guidance scale and condition masking.