Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation

Zengwei Yao, Wei Kang, Han Zhu, Liyong Guo, Lingxuan Ye, Fangjun Kuang, Weiji Zhuang, Zhaoqing Li, Zhifeng Han, Long Lin, Daniel Povey

TL;DR

Flow2GAN introduces a two-stage approach that fuses Flow Matching with GAN fine-tuning to achieve high-quality audio generation with only a few inference steps. It first trains an improved Flow Matching model tailored for audio by endpoint prediction and spectral energy-based loss scaling, then refines the generator via GAN training to enable one- to four-step inference. A novel multi-resolution ConvNeXt-based backbone processes Fourier coefficients at multiple time-frequency resolutions, enhancing modeling capacity. Empirical results on Mel-spectrogram and EnCodec token conditioning show Flow2GAN surpasses state-of-the-art GAN- and Flow Matching-based methods in quality-efficiency trade-offs, with strong speeds on CPU/GPU and favorable zero-shot TTS performance.

Abstract

Existing dominant methods for audio generation include Generative Adversarial Networks (GANs) and diffusion-based methods like Flow Matching. GANs suffer from slow convergence and potential mode collapse during training, while diffusion methods require multi-step inference that introduces considerable computational overhead. In this work, we introduce Flow2GAN, a two-stage framework that combines Flow Matching training for learning generative capabilities with GAN fine-tuning for efficient few-step inference. Specifically, given audio's unique properties, we first improve Flow Matching for audio modeling through: 1) reformulating the objective as endpoint estimation, avoiding velocity estimation difficulties when involving empty regions; 2) applying spectral energy-based loss scaling to emphasize perceptually salient quieter regions. Building on these Flow Matching adaptations, we demonstrate that a further stage of lightweight GAN fine-tuning enables us to obtain one-step generator that produces high-quality audio. In addition, we develop a multi-branch network architecture that processes Fourier coefficients at different time-frequency resolutions, which improves the modeling capabilities compared to prior single-resolution designs. Experimental results indicate that our Flow2GAN delivers high-fidelity audio generation from Mel-spectrograms or discrete audio tokens, achieving better quality-efficiency trade-offs than existing state-of-the-art GAN-based and Flow Matching-based methods. Online demo samples are available at https://flow2gan.github.io, and the source code is released at https://github.com/k2-fsa/Flow2GAN.

Flow2GAN: Hybrid Flow Matching and GAN with Multi-Resolution Network for Few-step High-Fidelity Audio Generation

TL;DR

Flow2GAN introduces a two-stage approach that fuses Flow Matching with GAN fine-tuning to achieve high-quality audio generation with only a few inference steps. It first trains an improved Flow Matching model tailored for audio by endpoint prediction and spectral energy-based loss scaling, then refines the generator via GAN training to enable one- to four-step inference. A novel multi-resolution ConvNeXt-based backbone processes Fourier coefficients at multiple time-frequency resolutions, enhancing modeling capacity. Empirical results on Mel-spectrogram and EnCodec token conditioning show Flow2GAN surpasses state-of-the-art GAN- and Flow Matching-based methods in quality-efficiency trade-offs, with strong speeds on CPU/GPU and favorable zero-shot TTS performance.

Abstract

Existing dominant methods for audio generation include Generative Adversarial Networks (GANs) and diffusion-based methods like Flow Matching. GANs suffer from slow convergence and potential mode collapse during training, while diffusion methods require multi-step inference that introduces considerable computational overhead. In this work, we introduce Flow2GAN, a two-stage framework that combines Flow Matching training for learning generative capabilities with GAN fine-tuning for efficient few-step inference. Specifically, given audio's unique properties, we first improve Flow Matching for audio modeling through: 1) reformulating the objective as endpoint estimation, avoiding velocity estimation difficulties when involving empty regions; 2) applying spectral energy-based loss scaling to emphasize perceptually salient quieter regions. Building on these Flow Matching adaptations, we demonstrate that a further stage of lightweight GAN fine-tuning enables us to obtain one-step generator that produces high-quality audio. In addition, we develop a multi-branch network architecture that processes Fourier coefficients at different time-frequency resolutions, which improves the modeling capabilities compared to prior single-resolution designs. Experimental results indicate that our Flow2GAN delivers high-fidelity audio generation from Mel-spectrograms or discrete audio tokens, achieving better quality-efficiency trade-offs than existing state-of-the-art GAN-based and Flow Matching-based methods. Online demo samples are available at https://flow2gan.github.io, and the source code is released at https://github.com/k2-fsa/Flow2GAN.
Paper Structure (19 sections, 6 equations, 3 figures, 11 tables)

This paper contains 19 sections, 6 equations, 3 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminary: Flow Matching
  4. Method
  5. Improved Flow Matching for audio modeling
  6. GAN Fine-tuning for Refinement and Few-step Inference
  7. Multi-resolution network structure
  8. Experiments
  9. Experimental Setup
  10. Comparison to state-of-the-art methods
  11. Ablation studies
  12. Inference speed
  13. Zero-shot TTS Performance Comparison
  14. Conclusion
  15. Appendix
  16. ...and 4 more sections

Figures (3)

  • Figure 1: Overall framework of Flow2GAN.
  • Figure 2: Example illustration of audio generation. Flow Matching estimates velocity $v_t=x_1-x_0$. The improved Flow Matching in Flow2GAN estimates endpoint $x_1$, yielding cleaner result in silent regions with 2 sampling steps. GAN fine-tuning further enhances the result by filling in details.
  • Figure 3: Multi-resolution network structure. Our model processes spectral coefficients from STFT at three different time-frequency resolutions, using larger embedding dimensions for shorter sequences. A ConvNeXt-based encoder processes the conditional compressed representation.