AudioStory: Generating Long-Form Narrative Audio with Large Language Models

TL;DR

AudioStory tackles the challenge of long-form narrative audio generation by coupling LLM-based planning with diffusion-based audio synthesis in an end-to-end framework. It introduces interleaved reasoning generation and a decoupled bridging mechanism with semantic and residual tokens, enabling temporally coherent scene transitions and consistent emotional tone. The paper also presents the AudioStory-10k benchmark and demonstrates significant gains over prior text-to-audio and unified models in both instruction-following and audio fidelity, with extensive ablations and human evaluations supporting the findings. The work advances practical long-form narrative audio generation and offers insights into end-to-end LLM–diffusion collaboration and bridging token design.

Abstract

Recent advances in text-to-audio (TTA) generation excel at synthesizing short audio clips but struggle with long-form narrative audio, which requires temporal coherence and compositional reasoning. To address this gap, we propose AudioStory, a unified framework that integrates large language models (LLMs) with TTA systems to generate structured, long-form audio narratives. AudioStory possesses strong instruction-following reasoning generation capabilities. It employs LLMs to decompose complex narrative queries into temporally ordered sub-tasks with contextual cues, enabling coherent scene transitions and emotional tone consistency. AudioStory has two appealing features: (1) Decoupled bridging mechanism: AudioStory disentangles LLM-diffuser collaboration into two specialized components, i.e., a bridging query for intra-event semantic alignment and a residual query for cross-event coherence preservation. (2) End-to-end training: By unifying instruction comprehension and audio generation within a single end-to-end framework, AudioStory eliminates the need for modular training pipelines while enhancing synergy between components. Furthermore, we establish a benchmark AudioStory-10K, encompassing diverse domains such as animated soundscapes and natural sound narratives. Extensive experiments show the superiority of AudioStory on both single-audio generation and narrative audio generation, surpassing prior TTA baselines in both instruction-following ability and audio fidelity. Our code is available at https://github.com/TencentARC/AudioStory

Figures (15)

• Figure 1: AudioStory effectively follows multimodal instructions, decomposing them into a sequence of coherent audio segments, capturing scene transitions, emotional tone, and segment timestamps. Unlike prior T5-based diffusion models, which struggle with complex queries, AudioStory empowers LLMs with high-level planning ability for instruction-followed and consistent long audio generation.
• Figure 2: Overview of AudioStory, with three core components: (a) A unified framewrok: The reasoning-capable LLM processes the instruction input, decomposes the long audio into structured generation sub-tasks, and sequentially generates a caption, semantic tokens, and residual tokens for each audio clip. (b) Audio Generator: After fusing semantic and residual tokens, they are combined with the duration information as conditioning inputs to the DiT, which then generates each audio clip. (c) Training strategy: Training is conducted in three stages to progressively enhance generation fidelity, semantic understanding, and global coherence.
• Figure 3: Qualitative case of long-form audio generation.
• Figure 4: Case of naive video dubbing: First, we extract captions from the video, then write the extracted captions as instructions and send them to AudioStory for audio generation.
• Figure 5: Case of video dubbing: We input both the video and the instruction into the model, which parses the narrative into segments, extracts story details with corresponding audio elements, and sequentially generates aligned audio clips..