Feedback-driven Retrieval-augmented Audio Generation with Large Audio Language Models

TL;DR

This work tackles missing or imperfect sound event generation in text-to-audio (TTA) by introducing a feedback-driven retrieval-augmented generation framework that leverages a Large Audio Language Model (LALM) to assess TTA outputs and identify gaps. A retrieval module based on CLAP fetches relevant audio prompts from unlabeled databases, and a lightweight audio fuser with decoupled cross-attention injects retrieved audio into pre-trained TTA models during inference, using a cross-attention fusion formula $z^{\mathrm{new}} = z_{t} + \lambda z_{a}$. The LALM is fine-tuned via LoRA to improve its identifications, and the system is validated across AudioCaps and RiTTA datasets with base models AudioLDM2-Large and TangoFlux, showing improvements over state-of-the-art RAG approaches without requiring training of a dedicated RAG model. Results indicate gains in both in-domain and out-of-domain scenarios, with TangoFlux-RAG achieving top performance on several metrics, highlighting the approach's broad applicability and cost-efficiency. The method promises practical impact by enabling more accurate, diverse, and context-aligned audio generation in real-world settings without extensive RAG-model training.

Abstract

We propose a general feedback-driven retrieval-augmented generation (RAG) approach that leverages Large Audio Language Models (LALMs) to address the missing or imperfect synthesis of specific sound events in text-to-audio (TTA) generation. Unlike previous RAG-based TTA methods that typically train specialized models from scratch, we utilize LALMs to analyze audio generation outputs, retrieve concepts that pre-trained models struggle to generate from an external database, and incorporate the retrieved information into the generation process. Experimental results show that our method not only enhances the ability of LALMs to identify missing sound events but also delivers improvements across different models, outperforming existing RAG-specialized approaches.

Figures (1)

• Figure 1: The overview structure of our proposed method.