Towards generalizing deep-audio fake detection networks

TL;DR

This work tackles the problem of generalizing deepfake-audio detectors to unseen speech generators by uncovering stable frequency-domain artifacts across modern synthetic voices. It builds a dilated CNN detector that operates on multiple frequency representations, notably wavelet-packet transforms and STFT, and demonstrates robust generalization to new generators such as Avocodo and BigVGAN. The authors extend the WaveFake dataset with these new models and show that detectors trained on a single generator can maintain high accuracy and low error rates on unfamiliar sources, outperforming prior WaveFake baselines. Interpretability is strengthened via Integrated Gradients, revealing a heavy influence of high-frequency components in distinguishing real from fake audio. The work provides practical, efficient detection methods and a dataset extension that keeps pace with rapid advances in audio synthesis, with implications for media integrity and platform security.

Abstract

Today's generative neural networks allow the creation of high-quality synthetic speech at scale. While we welcome the creative use of this new technology, we must also recognize the risks. As synthetic speech is abused for monetary and identity theft, we require a broad set of deepfake identification tools. Furthermore, previous work reported a limited ability of deep classifiers to generalize to unseen audio generators. We study the frequency domain fingerprints of current audio generators. Building on top of the discovered frequency footprints, we train excellent lightweight detectors that generalize. We report improved results on the WaveFake dataset and an extended version. To account for the rapid progress in the field, we extend the WaveFake dataset by additionally considering samples drawn from the novel Avocodo and BigVGAN networks. For illustration purposes, the supplementary material contains audio samples of generator artifacts.

Figures (21)

• Figure 1: Mean level 14 Haar-Wavelet decomposition of original LJSpeech (left) recordings as well as synthetic versions generated by Melgan (center). The difference between both plots is shown on the right. Melgan kumar2019melgan displays a characteristic spike-shaped fingerprint. Melgan produces characteristic spikes in the frequency domain.
• Figure 2: Mean level 14 Haar-Wavelet decomposition of original LJSpeech (left) recordings as well as synthetic versions generated by BigVgan (center). The plot on the right shows the difference. The more recent BigVGAN produces spikes in the spectral representation. Albeit less pronounced than those of MelGAN, BigVGAN's spectral representation diverges from the original LJSpeech coefficients, especially for higher frequencies.
• Figure 3: Structure of our dcnn. The Conv2d blocks denote 2D-Convolution operations with hyperparameters (Output Channels, Kernel Size, Padding, Dilation). We always work with unit strides. Each Conv2d is preceded by a Batch Normalization Layer ioffe2015batchnorm and followed by a PreLU activation xu2015prelu. The permutation operation permutes the first with the second dimension of the input (we consider the batch dimension to be dimension zero). $M$ denotes the number of output channels from the convolutional layers before.
• Figure 4: Attribution using integrated gradients on the-sym8-WPT-based dcnn-classifier evaluated over 2500 real audio samples from LJSpeech (left), 2500 fake audio samples from our extended version of the WaveFake dataset (middle) and averaged over both real and fake audios (right). We observe high values in the high-frequency domain as well as an inverse character between real and fake audio files. This WPT based cnn shows distinguishable interest regions for the model in several frequency bins.
• Figure 5: Attribution using integrated gradients sundararajan2017attribution on the STFT-dcnn-classifier for 2500 real audio samples from LJSpeech (left), 2500 fake audio samples from our extended version of the WaveFake dataset (middle) and averaged over both real and fake audios (right). We observe moderately high values in the high-frequency domain as well as an inverse character between real and fake audios similar to what we saw in Figure \ref{['fig:attribution_packets']}.