Robust AI-Synthesized Speech Detection Using Feature Decomposition Learning and Synthesizer Feature Augmentation

TL;DR

A robust deepfake speech detection method that employs feature decomposition to learn synthesizer-independent content features as complementary for detection and a synthesizer feature augmentation strategy that randomly blends the characteristic styles within real and fake audio features and randomly shuffles the synthesizer features with the content features is proposed.

Abstract

AI-synthesized speech, also known as deepfake speech, has recently raised significant concerns due to the rapid advancement of speech synthesis and speech conversion techniques. Previous works often rely on distinguishing synthesizer artifacts to identify deepfake speech. However, excessive reliance on these specific synthesizer artifacts may result in unsatisfactory performance when addressing speech signals created by unseen synthesizers. In this paper, we propose a robust deepfake speech detection method that employs feature decomposition to learn synthesizer-independent content features as complementary for detection. Specifically, we propose a dual-stream feature decomposition learning strategy that decomposes the learned speech representation using a synthesizer stream and a content stream. The synthesizer stream specializes in learning synthesizer features through supervised training with synthesizer labels. Meanwhile, the content stream focuses on learning synthesizer-independent content features, enabled by a pseudo-labeling-based supervised learning method. This method randomly transforms speech to generate speed and compression labels for training. Additionally, we employ an adversarial learning technique to reduce the synthesizer-related components in the content stream. The final classification is determined by concatenating the synthesizer and content features. To enhance the model's robustness to different synthesizer characteristics, we further propose a synthesizer feature augmentation strategy that randomly blends the characteristic styles within real and fake audio features and randomly shuffles the synthesizer features with the content features. This strategy effectively enhances the feature diversity and simulates more feature combinations.

Figures (5)

• Figure 1: Network architecture of our method. A main stream is used to learn robust speech representation from the log-scale frequency spectrogram of the input speech. Subsequently, a dual-stream learning strategy, comprising a synthesizer stream and a content stream, is employed to decompose the learned speech representation. The final classification is performed based on the concatenation of the synthesizer and content features. A synthesizer feature augmentation strategy consisting of feature blending and feature shuffle operations is employed to enhance the model's robustness to different synthesizer characteristics and synthesizer-content feature combinations.
• Figure 2: T-SNE visualization in the cross-evaluation task on the LibriseVoc dataset. For each deepfake speech detection method, we extract the latent features from the validation and test subsets and randomly extract 300 samples of the real and each fake method for visualization.
• Figure 3: T-SNE visualization in the cross-evaluation task on the DECRO ZH subset. For each deepfake speech detection method, we extract the latent features from the validation and test subsets and randomly extract 300 samples of the real and each fake method for visualization.
• Figure 4: Grad-CAM visualization on the LibriseVoc dataset. From top to bottom of each column, the three images are raw log-scale spectrograms, gradient visualizations from the synthesizer, and content features. The brighter color denotes a larger influence on the classification. Note all the used speech samples are deepfake from the DiffWave subset, and the column names denote the file names.
• Figure 5: Ablation results (EER (%)) of the contrastive losses on two ablation tasks.