Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fast Text-to-Audio Generation with Adversarial Post-Training

Zachary Novack, Zach Evans, Zack Zukowski, Josiah Taylor, CJ Carr, Julian Parker, Adnan Al-Sinan, Gian Marco Iodice, Julian McAuley, Taylor Berg-Kirkpatrick, Jordi Pons

TL;DR

This work tackles the latency of text-to-audio diffusion models by introducing Adversarial Relativistic-Contrastive post-training (ARC), a non-distillation acceleration framework that couples a relativistic adversarial loss with a contrastive discriminator objective. ARC leverages rectified-flow pre-training, ping-pong sampling, and a reward-model–like interpretation to produce high-fidelity audio in far fewer steps, achieving ~12 seconds of 44.1 kHz stereo audio in ~75 ms on an H100 and ~7 seconds on edge devices, while preserving diversity better than prior methods. The authors validate ARC against strong baselines, perform comprehensive ablations, and demonstrate edge-device deployment with significant RAM and latency reductions, enabling practical on-device text-to-audio generation and creative workflows. The approach promises faster, more diverse audio generation without distillation or CFG, opening avenues for real-time sound design and on-device applications.

Abstract

Text-to-audio systems, while increasingly performant, are slow at inference time, thus making their latency unpractical for many creative applications. We present Adversarial Relativistic-Contrastive (ARC) post-training, the first adversarial acceleration algorithm for diffusion/flow models not based on distillation. While past adversarial post-training methods have struggled to compare against their expensive distillation counterparts, ARC post-training is a simple procedure that (1) extends a recent relativistic adversarial formulation to diffusion/flow post-training and (2) combines it with a novel contrastive discriminator objective to encourage better prompt adherence. We pair ARC post-training with a number optimizations to Stable Audio Open and build a model capable of generating $\approx$12s of 44.1kHz stereo audio in $\approx$75ms on an H100, and $\approx$7s on a mobile edge-device, the fastest text-to-audio model to our knowledge.

Fast Text-to-Audio Generation with Adversarial Post-Training

TL;DR

This work tackles the latency of text-to-audio diffusion models by introducing Adversarial Relativistic-Contrastive post-training (ARC), a non-distillation acceleration framework that couples a relativistic adversarial loss with a contrastive discriminator objective. ARC leverages rectified-flow pre-training, ping-pong sampling, and a reward-model–like interpretation to produce high-fidelity audio in far fewer steps, achieving ~12 seconds of 44.1 kHz stereo audio in ~75 ms on an H100 and ~7 seconds on edge devices, while preserving diversity better than prior methods. The authors validate ARC against strong baselines, perform comprehensive ablations, and demonstrate edge-device deployment with significant RAM and latency reductions, enabling practical on-device text-to-audio generation and creative workflows. The approach promises faster, more diverse audio generation without distillation or CFG, opening avenues for real-time sound design and on-device applications.

Abstract

Text-to-audio systems, while increasingly performant, are slow at inference time, thus making their latency unpractical for many creative applications. We present Adversarial Relativistic-Contrastive (ARC) post-training, the first adversarial acceleration algorithm for diffusion/flow models not based on distillation. While past adversarial post-training methods have struggled to compare against their expensive distillation counterparts, ARC post-training is a simple procedure that (1) extends a recent relativistic adversarial formulation to diffusion/flow post-training and (2) combines it with a novel contrastive discriminator objective to encourage better prompt adherence. We pair ARC post-training with a number optimizations to Stable Audio Open and build a model capable of generating 12s of 44.1kHz stereo audio in 75ms on an H100, and 7s on a mobile edge-device, the fastest text-to-audio model to our knowledge.
Paper Structure (18 sections, 6 equations, 2 figures, 1 table)

This paper contains 18 sections, 6 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. METHOD
  3. Rectified flows pre-training
  4. Adversarial Relativistic-Contrastive post-training (ARC)
  5. Adversarial relativistic loss: the AR in ARC
  6. Contrastive loss: the C in ARC
  7. Ping-pong sampling
  8. Acceleration as reward modeling: a post-training perspective
  9. Experiments
  10. Models
  11. Training and sampling details
  12. Objective Evalaution
  13. Subjective Evaluation
  14. Baselines
  15. Results and discussion
  16. ...and 3 more sections

Figures (2)

  • Figure 1: ARC's adversarial relativistic loss. Pairs of generated and real samples (with the same text prompts) are passed into the discriminator (with additive noise), where the generator and discriminator are trained to minimize and maximize (respectively) the difference between fake and real outputs.
  • Figure 2: ARC's contrastive loss. The discriminator is also trained to maximize the difference between audios with correct and incorrect (shuffled) prompts.