Addressing Gradient Misalignment in Data-Augmented Training for Robust Speech Deepfake Detection

TL;DR

This work investigates gradient misalignment arising from data augmentation in speech deepfake detection and introduces a dual-path data-augmented (DPDA) framework that processes both original and augmented inputs per utterance. By applying gradient alignment methods—PCGrad, GradVac, and CAGrad—the approach mitigates conflicting updates and accelerates convergence, improving robustness across multiple architectures and augmentation strategies. Empirically, about $25\%$ of training iterations in DPDA exhibit gradient conflicts with RawBoost, and alignment reduces training epochs to convergence while delivering up to a $18.69\%$ relative reduction in $EER$ on In-the-Wild, with consistent gains across datasets (21DF, ITW, FoR) and architectures. The results indicate that gradient alignment is a practical, architecture-agnostic enhancement for data-augmented SDD, guiding models to emphasize spoof-related cues over augmentation artifacts.

Abstract

In speech deepfake detection (SDD), data augmentation (DA) is commonly used to improve model generalization across varied speech conditions and spoofing attacks. However, during training, the backpropagated gradients from original and augmented inputs may misalign, which can result in conflicting parameter updates. These conflicts could hinder convergence and push the model toward suboptimal solutions, thereby reducing the benefits of DA. To investigate and address this issue, we design a dual-path data-augmented (DPDA) training framework with gradient alignment for SDD. In our framework, each training utterance is processed through two input paths: one using the original speech and the other with its augmented version. This design allows us to compare and align their backpropagated gradient directions to reduce optimization conflicts. Our analysis shows that approximately 25% of training iterations exhibit gradient conflicts between the original inputs and their augmented counterparts when using RawBoost augmentation. By resolving these conflicts with gradient alignment, our method accelerates convergence by reducing the number of training epochs and achieves up to an 18.69% relative reduction in Equal Error Rate on the In-the-Wild dataset compared to the baseline.

Figures (4)

• Figure 1: Overview of the dual-path data-augmented (DPDA) training framework with gradient alignment for SDD. The left panel illustrates the overall training framework, wherein both original and augmented inputs are processed in parallel. The right panel compares three gradient alignment methods, including PCGrad pcgrad, GradVac gradvac, and CAGrad cagrad, which are used to resolve gradient conflicts.
• Figure 2: Training loss and backpropagated gradient norm of orignal $x$ and augmented $\tilde{x}$ inputs during the DPDA training of XLSR-Conformer-TCM model.
• Figure 3: Loss surface visualization for original and augmented inputs of the XLSR-Conformer-TCM model during mid-training
• Figure 4: Number of gradient conflicts and validation loss during DPDA training of the XLSR-Conformer-TCM model, with and without the PCGrad gradient alignment method. Each epoch contains 5,076 iterations.