Generative Audio Extension and Morphing

TL;DR

Masking the noisy latents of a DiT and applying a novel variant of classifier-free guidance on such masked latents demonstrates that given an audio reference, the model can extend it both forward and backward for a specified duration, and given two audio references, it can morph them seamlessly for the desired duration.

Abstract

In audio-related creative tasks, sound designers often seek to extend and morph different sounds from their libraries. Generative audio models, capable of creating audio using examples as references, offer promising solutions. By masking the noisy latents of a DiT and applying a novel variant of classifier-free guidance on such masked latents, we demonstrate that: (i) given an audio reference, we can extend it both forward and backward for a specified duration, and (ii) given two audio references, we can morph them seamlessly for the desired duration. Furthermore, we show that by fine-tuning the model on different types of stationary audio data we mitigate potential hallucinations. The effectiveness of our method is supported by objective metrics, with the generated audio achieving Fréchet Audio Distances (FADs) comparable to those of real samples from the training data. Additionally, we validate our results through a subjective listener test, where subjects gave positive ratings to the proposed model generations. This technique paves the way for more controllable and expressive generative sound frameworks, enabling sound designers to focus less on tedious, repetitive tasks and more on their actual creative process.

Figures (2)

• Figure 1: Proposed block diagram of Generative Extend (solid lines) and Morphing (dashed lines). Two audio examples are shown: the blue one is used for Generative Extend (both forward and backward) and both the blue and the green ones are used for Morphing (from blue to green).
• Figure 2: Ablation of the Audio Prompt Guidance technique.