SemanticAudio: Audio Generation and Editing in Semantic Space

TL;DR

SemanticAudio addresses semantic misalignment in text-to-audio generation by decoupling high-level semantic planning from acoustic rendering and operating in a compact high-level semantic space. It introduces a two-stage Flow Matching framework with a Semantic Planner that maps text to a semantic latent of dimension $d$, and an Acoustic Synthesizer that renders high-fidelity acoustics conditioned on that plan, along with a training-free editing mechanism based on the difference of velocity fields. The approach, evaluated on AudioCaps, achieves state-of-the-art semantic alignment (CLAP) and demonstrates robust, inversion-free editing without additional training. This decoupled, controllable paradigm advances practical text-to-audio generation and editing, with potential for longer-form audio and multi-modal controls.

Abstract

In recent years, Text-to-Audio Generation has achieved remarkable progress, offering sound creators powerful tools to transform textual inspirations into vivid audio. However, existing models predominantly operate directly in the acoustic latent space of a Variational Autoencoder (VAE), often leading to suboptimal alignment between generated audio and textual descriptions. In this paper, we introduce SemanticAudio, a novel framework that conducts both audio generation and editing directly in a high-level semantic space. We define this semantic space as a compact representation capturing the global identity and temporal sequence of sound events, distinct from fine-grained acoustic details. SemanticAudio employs a two-stage Flow Matching architecture: the Semantic Planner first generates these compact semantic features to sketch the global semantic layout, and the Acoustic Synthesizer subsequently produces high-fidelity acoustic latents conditioned on this semantic plan. Leveraging this decoupled design, we further introduce a training-free text-guided editing mechanism that enables precise attribute-level modifications on general audio without retraining. Specifically, this is achieved by steering the semantic generation trajectory via the difference of velocity fields derived from source and target text prompts. Extensive experiments demonstrate that SemanticAudio surpasses existing mainstream approaches in semantic alignment. Demo available at: https://semanticaudio1.github.io/

Figures (2)

• Figure 1: Overview of the SemanticAudio framework. The model employs a two-stage Flow Matching architecture: the Semantic Planner first generates low-dimensional semantic latents conditioned on text, followed by the Acoustic Synthesizer which produces high-fidelity acoustic latents for VAE decoding.
• Figure 2: Overview of our training-free text-guided audio editing method. The process leverages the pre-trained velocity fields of the Semantic Planner to perform semantic-level editing in the low-dimensional latent space via difference velocity integration, followed by high-fidelity reconstruction using the Acoustic Synthesizer. The method requires no additional training, inversion, or optimization.