DegDiT: Controllable Audio Generation with Dynamic Event Graph Guided Diffusion Transformer

TL;DR

DegDiT addresses the challenge of open-vocabulary, fine-grained controllable text-to-audio generation by representing multi-event prompts as temporally dynamic graphs $G=(V,E)$ and guiding diffusion with a graph transformer to achieve precise content and timing. It introduces a Quality-Balanced Data Selection pipeline to curate diverse, high-quality training data and Consensus Preference Optimization (CoPO) to fuse multiple reward signals for training. Empirical results on AudioCondition, DESED, and AudioTime show state-of-the-art performance in temporal precision, semantic alignment, and audio quality, with strong subjective ratings. The framework enables robust, open-vocabulary audio generation and highlights future work on improving minority-event handling via larger timestamp-annotated datasets.

Abstract

Controllable text-to-audio generation aims to synthesize audio from textual descriptions while satisfying user-specified constraints, including event types, temporal sequences, and onset and offset timestamps. This enables precise control over both the content and temporal structure of the generated audio. Despite recent progress, existing methods still face inherent trade-offs among accurate temporal localization, open-vocabulary scalability, and practical efficiency. To address these challenges, we propose DegDiT, a novel dynamic event graph-guided diffusion transformer framework for open-vocabulary controllable audio generation. DegDiT encodes the events in the description as structured dynamic graphs. The nodes in each graph are designed to represent three aspects: semantic features, temporal attributes, and inter-event connections. A graph transformer is employed to integrate these nodes and produce contextualized event embeddings that serve as guidance for the diffusion model. To ensure high-quality and diverse training data, we introduce a quality-balanced data selection pipeline that combines hierarchical event annotation with multi-criteria quality scoring, resulting in a curated dataset with semantic diversity. Furthermore, we present consensus preference optimization, facilitating audio generation through consensus among multiple reward signals. Extensive experiments on AudioCondition, DESED, and AudioTime datasets demonstrate that DegDiT achieves state-of-the-art performances across a variety of objective and subjective evaluation metrics.

Figures (4)

• Figure 1: Comparison between naive text-only guided (left) and our proposed dynamic event graph guided (right) diffusion model generation. The naive text-only guided approach relies solely on textual input to drive the diffusion process. By integrating structured event graphs, our approach effectively captures temporal dependencies, enabling the generation of accurate and consistent timestamps.
• Figure 2: The DegDiT overall framework decouples events from their timestamps and constructs frame-level event and relation matrices, which are subsequently mapped into structured graphs via dedicated encoders. A transformer aggregates these graphs to capture contextual dependencies, while a multi-modal diffusion transformer synthesizes output audio conditioned on both textual prompts and the structured graphs. The training pipeline is optimized with the flow matching loss followed by CoPO across multiple reward signals.
• Figure 3: We provide Gemini 2.5 Pro with the evaluation template shown above, including overall descriptions and expected temporal segments (annotated from the ground truth), and request ratings across four dimensions: content matching, timing accuracy, audio quality, and overall coherence.
• Figure 4: We present examples of prompts containing single events, two events, multiple events, and overlapping events (in columns) for Tangoflux, Audiocomposer, and our proposed DegDiT (in rows), respectively.