Region-Based Optimization in Continual Learning for Audio Deepfake Detection

TL;DR

This work tackles continual learning for robust audio deepfake detection amid diverse and evolving forgeries. It introduces Region-Based Optimization (RegO), which uses the Fisher information matrix to partition network parameters into four importance regions and applies region-specific gradient updates, augmented by an Ebbinghaus-inspired forgetting mechanism to shed redundant neurons. On the EVDA benchmark, RegO substantially outperforms standard continual-learning baselines like RAWM and RWM and nears the Replay-All upper bound, with a reported EER improvement of about 21.3% over RWM. A cross-domain exploration in image recognition (CLEAR) further indicates RegO's potential applicability beyond audio, highlighting its generality for balancing stability and plasticity in continual learning tasks.

Abstract

Rapid advancements in speech synthesis and voice conversion bring convenience but also new security risks, creating an urgent need for effective audio deepfake detection. Although current models perform well, their effectiveness diminishes when confronted with the diverse and evolving nature of real-world deepfakes. To address this issue, we propose a continual learning method named Region-Based Optimization (RegO) for audio deepfake detection. Specifically, we use the Fisher information matrix to measure important neuron regions for real and fake audio detection, dividing them into four regions. First, we directly fine-tune the less important regions to quickly adapt to new tasks. Next, we apply gradient optimization in parallel for regions important only to real audio detection, and in orthogonal directions for regions important only to fake audio detection. For regions that are important to both, we use sample proportion-based adaptive gradient optimization. This region-adaptive optimization ensures an appropriate trade-off between memory stability and learning plasticity. Additionally, to address the increase of redundant neurons from old tasks, we further introduce the Ebbinghaus forgetting mechanism to release them, thereby promoting the capability of the model to learn more generalized discriminative features. Experimental results show our method achieves a 21.3% improvement in EER over the state-of-the-art continual learning approach RWM for audio deepfake detection. Moreover, the effectiveness of RegO extends beyond the audio deepfake detection domain, showing potential significance in other tasks, such as image recognition. The code is available at https://github.com/cyjie429/RegO

Figures (2)

• Figure 1: Illustration of RegO Architecture. (i).After the training of each task, we calculate the region matrix $\mathbf{R}$ through the IRL module. (ii).From the second task onward, gradient optimization is performed during backpropagation (shown by dark red arrows and boxes). The $\mathbf{E}_k$ is obtained through the EFM module and then combined with the historical $\mathbf{R}$ to generate $\overline{\mathbf{R}}$. (iii).Based on $\overline{\mathbf{R}}$, the RAO module updates the model weights as follows: Region A: fine-tuning (i.e. $g$); Region B: gradient update in the projection direction (i.e. $g_p$); Region C: gradient update in the orthogonal direction (i.e. $g_o$); Region D: adaptive gradient update based on the number of samples (i.e. $\widetilde{g}$).
• Figure 2: The average EER (%) results of the hyperparameter study for our method RegO.