Retrieval-Augmented Text-to-Audio Generation

TL;DR

This work tackles the long-tailed bias in text-to-audio generation caused by imbalanced training data. It introduces Re-AudioLDM, a retrieval-augmented diffusion framework that uses retrieved text-audio pairs via CLAP, AudioMAE, and T5 embeddings to condition generation through cross-attention, guided by a $L_{n}(\theta)$ objective and a VAE–HiFi-GAN decoding pipeline. On AudioCaps, Re-AudioLDM achieves state-of-the-art Fréchet Audio Distance and substantially improves tail- and unseen-event generation, with ablations showing that combining text and audio retrieval yields the strongest gains and that 3–5 retrieved items balance performance and cost. The results indicate strong potential for robust, realistic TTA in diverse acoustic scenarios and open avenues for zero-shot generation and broader retrieval-enhanced audio synthesis.

Abstract

Despite recent progress in text-to-audio (TTA) generation, we show that the state-of-the-art models, such as AudioLDM, trained on datasets with an imbalanced class distribution, such as AudioCaps, are biased in their generation performance. Specifically, they excel in generating common audio classes while underperforming in the rare ones, thus degrading the overall generation performance. We refer to this problem as long-tailed text-to-audio generation. To address this issue, we propose a simple retrieval-augmented approach for TTA models. Specifically, given an input text prompt, we first leverage a Contrastive Language Audio Pretraining (CLAP) model to retrieve relevant text-audio pairs. The features of the retrieved audio-text data are then used as additional conditions to guide the learning of TTA models. We enhance AudioLDM with our proposed approach and denote the resulting augmented system as Re-AudioLDM. On the AudioCaps dataset, Re-AudioLDM achieves a state-of-the-art Frechet Audio Distance (FAD) of 1.37, outperforming the existing approaches by a large margin. Furthermore, we show that Re-AudioLDM can generate realistic audio for complex scenes, rare audio classes, and even unseen audio types, indicating its potential in TTA tasks.

Figures (4)

• Figure 1: The long-tailed problem in AudioCaps dataset (top). Example audio clips (bottom) generated with the baseline model (left) and Re-AudioLDM (right).
• Figure 2: The overview structure of Re-AudioLDM
• Figure 3: Performance comparison on numbers of retrieved information, where AC_cap 1-5 refers to the caption groups of the testing set.
• Figure 4: Performance on different frequency entities, where S and L indicate model size and r refers to the number of retrieved clips.