Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complex Image-Generative Diffusion Transformer for Audio Denoising

Junhui Li, Pu Wang, Jialu Li, Youshan Zhang

TL;DR

This work tackles audio denoising by reframing the task as a complex image-generation problem in the Fourier domain and introducing a complex image-generative diffusion transformer (CIGDTN). The model fuses diffusion-transformer principles with adaptive layer normalization (adaLN-Zero) and sparse attention diffusion to process real and imaginary spectrograms, enabling richer complex-valued representations via a complex encoder, CIGDT blocks, and a decoder, with final ISTFT reconstruction. A multi-input training objective combines energy-aware image losses and time-domain SDR-oriented loss, and the authors employ FlashAttention-2 for efficiency. Experimental results on VoiceBank+DEMAND and BirdSoundsDenoising demonstrate state-of-the-art or competitive performance across standard perceptual and signal-level metrics, validating the approach's effectiveness and scalability. Overall, the paper advances audio denoising by leveraging complex-valued diffusion-transformer architectures to exploit phase information and extended receptive fields, with practical implications for high-quality speech and environmental audio enhancement.

Abstract

The audio denoising technique has captured widespread attention in the deep neural network field. Recently, the audio denoising problem has been converted into an image generation task, and deep learning-based approaches have been applied to tackle this problem. However, its performance is still limited, leaving room for further improvement. In order to enhance audio denoising performance, this paper introduces a complex image-generative diffusion transformer that captures more information from the complex Fourier domain. We explore a novel diffusion transformer by integrating the transformer with a diffusion model. Our proposed model demonstrates the scalability of the transformer and expands the receptive field of sparse attention using attention diffusion. Our work is among the first to utilize diffusion transformers to deal with the image generation task for audio denoising. Extensive experiments on two benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods.

Complex Image-Generative Diffusion Transformer for Audio Denoising

TL;DR

This work tackles audio denoising by reframing the task as a complex image-generation problem in the Fourier domain and introducing a complex image-generative diffusion transformer (CIGDTN). The model fuses diffusion-transformer principles with adaptive layer normalization (adaLN-Zero) and sparse attention diffusion to process real and imaginary spectrograms, enabling richer complex-valued representations via a complex encoder, CIGDT blocks, and a decoder, with final ISTFT reconstruction. A multi-input training objective combines energy-aware image losses and time-domain SDR-oriented loss, and the authors employ FlashAttention-2 for efficiency. Experimental results on VoiceBank+DEMAND and BirdSoundsDenoising demonstrate state-of-the-art or competitive performance across standard perceptual and signal-level metrics, validating the approach's effectiveness and scalability. Overall, the paper advances audio denoising by leveraging complex-valued diffusion-transformer architectures to exploit phase information and extended receptive fields, with practical implications for high-quality speech and environmental audio enhancement.

Abstract

The audio denoising technique has captured widespread attention in the deep neural network field. Recently, the audio denoising problem has been converted into an image generation task, and deep learning-based approaches have been applied to tackle this problem. However, its performance is still limited, leaving room for further improvement. In order to enhance audio denoising performance, this paper introduces a complex image-generative diffusion transformer that captures more information from the complex Fourier domain. We explore a novel diffusion transformer by integrating the transformer with a diffusion model. Our proposed model demonstrates the scalability of the transformer and expands the receptive field of sparse attention using attention diffusion. Our work is among the first to utilize diffusion transformers to deal with the image generation task for audio denoising. Extensive experiments on two benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods.
Paper Structure (12 sections, 10 equations, 2 figures, 2 tables, 1 algorithm)

This paper contains 12 sections, 10 equations, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. CIGDTN Model
  4. Complex Encoder
  5. CIGDT Block
  6. Decoder
  7. Objective Function
  8. Experiment
  9. Datasets
  10. Evaluation Metrics
  11. Result
  12. Conclusion

Figures (2)

  • Figure 1: A overall progress of our proposed CIGDTN model. Different modules are marked with different color blocks. The architecture of the main body fuses diffusion transformers (DiTs) architecture and sparse attention diffusion. We first apply STFT to convert audio signals $Y$ into complex images (real image $X^{real}_{k,f}$ and imaginary image $X^{imag}_{k,f}$). Then, we feed them into the model, which generates real image $M(X^{real}_{k,f})$ and imaginary image $M(X^{imag}_{k,f})$. Finally, we reconstruct clean audio using ISTFT.
  • Figure 2: CIGDT block architecture. Each block is composed of a complex Multi-Head Self-Attention (MSA) module and sparse attention diffusion, and a complex feed-forward layer including a complex LayerNorm layer, a two-fully connected layers complex MLP (CMLP). A scale and shift layer is located behind the LayerNorm layer.