Audio Texture Manipulation by Exemplar-Based Analogy

TL;DR

The paper addresses the challenge of editing audio textures when natural language prompts are ambiguous by introducing exemplar-based analogy, where an input is transformed according to a demonstrated before/after exemplar pair. It wires a latent diffusion model conditioned on exemplar embeddings, operating in a VAE-compressed mel-spectrogram space and reconstructed via a HiFi-GAN vocoder, with learnable temporal encoding and cross-attention to guide transformations. A self-supervised quadruplet dataset combines LibriSpeech/VCTK with BBC SFX to learn add/remove/replace transformations without labeled edits, achieving competitive or better results than text-conditioned baselines and generalizing to real-world and non-speech data. This exemplar-driven framework offers a flexible, intuitive approach for audio editing with potential applicability beyond speech to broader sound textures and real-world audio manipulation.

Abstract

Audio texture manipulation involves modifying the perceptual characteristics of a sound to achieve specific transformations, such as adding, removing, or replacing auditory elements. In this paper, we propose an exemplar-based analogy model for audio texture manipulation. Instead of conditioning on text-based instructions, our method uses paired speech examples, where one clip represents the original sound and another illustrates the desired transformation. The model learns to apply the same transformation to new input, allowing for the manipulation of sound textures. We construct a quadruplet dataset representing various editing tasks, and train a latent diffusion model in a self-supervised manner. We show through quantitative evaluations and perceptual studies that our model outperforms text-conditioned baselines and generalizes to real-world, out-of-distribution, and non-speech scenarios. Project page: https://berkeley-speech-group.github.io/audio-texture-analogy/

Figures (4)

• Figure 1: Exemplar-based analogy for audio texture manipulation. We manipulate input speech (middle) based on an exemplar pair (left), where the pair defines the desired transformation such as adding, removing, or replacing specific sound elements.
• Figure 2: Model architecture. Given the input audio and exemplar pair, our goal is to transform the input to match the texture transformation demonstrated by the exemplar pair. We employ a pre-trained VAE encoder to encode both the input and target spectrograms to the latent space, and feed them into a latent diffusion model together with the exemplar pair embedding and positional encoding. Finally, we use pre-trained VAE decoder and HiFi-GAN vocoder to reconstruct the waveform from the latent space. Note that the VAE encoder for the target spectrogram is not used at test time.
• Figure 3: Model comparison. We present qualitative results between our model and AUDIT, where each input audio is transformed according to the exemplar pairs.
• Figure 4: Generalization to real-world data. Our model can generalize to non-speech (top) and real-world (bottom) scenarios.