LiteFocus: Accelerated Diffusion Inference for Long Audio Synthesis

TL;DR

This work tackles the inefficiency of diffusion-based text-to-audio models for long-form audio due to $O(N^2)$ self-attention costs. It introduces LiteFocus, a dual sparse attention mechanism that partitions attention into Same-frequency Focusing and Cross-frequency Compensation, forming $F_i = S_i ∪ C$ with $S_i = \{j | j \text{ mod } N_t = i \text{ mod } N_t\}$ and $C = RandomSample(I, \text{floor}(r|I|))$, enabling dual sparse attention. The approach accelerates inference by up to 1.99x on 80-second audio while preserving or improving quality metrics on a standardized AudioCap set, without retraining. Ablation results show same-frequency focusing benefits long-audio quality and cross-frequency compensation benefits short audio, demonstrating the value of length-aware sparsity for diffusion-based long-form synthesis.

Abstract

Latent diffusion models have shown promising results in audio generation, making notable advancements over traditional methods. However, their performance, while impressive with short audio clips, faces challenges when extended to longer audio sequences. These challenges are due to model's self-attention mechanism and training predominantly on 10-second clips, which complicates the extension to longer audio without adaptation. In response to these issues, we introduce a novel approach, LiteFocus that enhances the inference of existing audio latent diffusion models in long audio synthesis. Observed the attention pattern in self-attention, we employ a dual sparse form for attention calculation, designated as same-frequency focus and cross-frequency compensation, which curtails the attention computation under same-frequency constraints, while enhancing audio quality through cross-frequency refillment. LiteFocus demonstrates substantial reduction on inference time with diffusion-based TTA model by 1.99x in synthesizing 80-second audio clips while also obtaining improved audio quality.

Figures (3)

• Figure 1: Illustration of the computational cost associated with audio synthesis using the audio diffusion model (AudioLDM2): (a) The time required for synthesizing audio clips of varying lengths; (b) The proportional time consumption of different modules within the model during the generation of audio clips of different lengths.
• Figure 2: Attention patterns within the audio latent diffusion model: (a) Attentions for two different tokens. We reshape the attentions to rearrange the attention of the same frequency in the same row. (b) Attention patterns across different blocks in the model. Due to size constraints, only a proportion of the attention maps are shown.
• Figure 3: A comparison between the original attention processor and LiteFocus: The left side illustrates the processor for the original self-attention mechanism, while the right side depicts the attention processor utilized by LiteFocus. In LiteFocus , each query is assigned a specific focus tokens set, with which it performs the attention operation on the corresponding keys and values.