SAO-Instruct: Free-form Audio Editing using Natural Language Instructions

TL;DR

SAO-Instruct presents the first fully free-form, instruction-based audio editing model built on Stable Audio Open. It combines three data-generation pipelines—Prompt-to-Prompt, DDPM inversion, and manual edits—with LLM-driven prompt synthesis to train on a large, diverse set of audio-edit triplets. The approach yields competitive objective metrics and superior subjective edit quality, while generalizing to real-world audio and maintaining input context. The authors release code and weights to foster further research in flexible, natural-language audio editing and editing workflows.

Abstract

Generative models have made significant progress in synthesizing high-fidelity audio from short textual descriptions. However, editing existing audio using natural language has remained largely underexplored. Current approaches either require the complete description of the edited audio or are constrained to predefined edit instructions that lack flexibility. In this work, we introduce SAO-Instruct, a model based on Stable Audio Open capable of editing audio clips using any free-form natural language instruction. To train our model, we create a dataset of audio editing triplets (input audio, edit instruction, output audio) using Prompt-to-Prompt, DDPM inversion, and a manual editing pipeline. Although partially trained on synthetic data, our model generalizes well to real in-the-wild audio clips and unseen edit instructions. We demonstrate that SAO-Instruct achieves competitive performance on objective metrics and outperforms other audio editing approaches in a subjective listening study. To encourage future research, we release our code and model weights.

Figures (8)

• Figure 1: Given an input audio clip and an edit instruction, SAO-Instruct outputs the edited audio while keeping the overall audio context intact.
• Figure 2: Overview of our proposed method. Green indicates synthetic data. Audio datasets are used as the starting point for prompt generation. DDPM inversion and Prompt-to-Prompt use the input caption and generated output caption to create a partial and fully synthetic dataset, respectively. For manual edits, a predefined edit operation is sampled. In the fine-tuning stage, Stable Audio Open is trained on the combined generated samples and edit instructions. During inference, SAO-Instruct receives an audio clip and a free-form edit instruction and produces the edited output.
• Figure 3: Pipeline for Prompt-to-Prompt audio generation. In (a), various seeds and CFG value combinations are tested and filtered using Gemini and CLAP to identify suitable configurations for prompts. In (b) Stable Audio Open (SAO) with Prompt-to-Prompt generates audio pairs using the seed and CFG configuration found in (a). A Bayesian Optimization process suggests Prompt-to-Prompt parameters and resulting samples are evaluated using an objective function. For clarity, only 3 candidate pairs and 2 Bayesian Optimization trials are shown.
• Figure 4: Pipeline for prompt generation. A caption is taken from a dataset and passed to an LLM, which produces an edit instruction and a corresponding output caption. Additional metadata is generated for downstream filtering and for improving sample quality for synthetic audio generation.
• Figure 5: Evaluation interface for the subjective listening study comparing SAO-Instruct with audio editing baselines.