Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio

Siding Zeng, Jiangyan Yi, Jianhua Tao, Yujie Chen, Shan Liang, Yong Ren, Xiaohui Zhang

TL;DR

This work tackles cross-domain localization of manipulated regions in partially fake audio by introducing Samples mining with Diversity and Entropy (SDE), an unsupervised domain adaptation framework. It combines reverse knowledge distillation to train diverse experts, entropy-based sampling to select informative target-domain samples, and a label generation process that swaps voice segments to inject target-domain information into source-domain training. The approach achieves a 43.84% F1 on ADD2023 Track2 with only 10% target-domain data, corresponding to a 77.2% improvement over the second-best method, and ablations confirm the value of diversity, sampling strategy, and unlabeled data labeling. The method advances cross-domain PF localization and suggests entropy-based sample mining as a broadly applicable technique for domain shifts.

Abstract

When the task of locating manipulation regions in partially-fake audio (PFA) involves cross-domain datasets, the performance of deep learning models drops significantly due to the shift between the source and target domains. To address this issue, existing approaches often employ data augmentation before training. However, they overlook the characteristics in target domain that are absent in source domain. Inspired by the mixture-of-experts model, we propose an unsupervised method named Samples mining with Diversity and Entropy (SDE). Our method first learns from a collection of diverse experts that achieve great performance from different perspectives in the source domain, but with ambiguity on target samples. We leverage these diverse experts to select the most informative samples by calculating their entropy. Furthermore, we introduced a label generation method tailored for these selected samples that are incorporated in the training process in source domain integrating the target domain information. We applied our method to a cross-domain partially fake audio detection dataset, ADD2023Track2. By introducing 10% of unknown samples from the target domain, we achieved an F1 score of 43.84%, which represents a relative increase of 77.2% compared to the second-best method.

An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio

TL;DR

This work tackles cross-domain localization of manipulated regions in partially fake audio by introducing Samples mining with Diversity and Entropy (SDE), an unsupervised domain adaptation framework. It combines reverse knowledge distillation to train diverse experts, entropy-based sampling to select informative target-domain samples, and a label generation process that swaps voice segments to inject target-domain information into source-domain training. The approach achieves a 43.84% F1 on ADD2023 Track2 with only 10% target-domain data, corresponding to a 77.2% improvement over the second-best method, and ablations confirm the value of diversity, sampling strategy, and unlabeled data labeling. The method advances cross-domain PF localization and suggests entropy-based sample mining as a broadly applicable technique for domain shifts.

Abstract

When the task of locating manipulation regions in partially-fake audio (PFA) involves cross-domain datasets, the performance of deep learning models drops significantly due to the shift between the source and target domains. To address this issue, existing approaches often employ data augmentation before training. However, they overlook the characteristics in target domain that are absent in source domain. Inspired by the mixture-of-experts model, we propose an unsupervised method named Samples mining with Diversity and Entropy (SDE). Our method first learns from a collection of diverse experts that achieve great performance from different perspectives in the source domain, but with ambiguity on target samples. We leverage these diverse experts to select the most informative samples by calculating their entropy. Furthermore, we introduced a label generation method tailored for these selected samples that are incorporated in the training process in source domain integrating the target domain information. We applied our method to a cross-domain partially fake audio detection dataset, ADD2023Track2. By introducing 10% of unknown samples from the target domain, we achieved an F1 score of 43.84%, which represents a relative increase of 77.2% compared to the second-best method.
Paper Structure (19 sections, 10 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 19 sections, 10 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. related work
  3. proposed method
  4. Model Always with Ambiguity
  5. Comprehensive Coverage
  6. Training Diverse Experts
  7. Selecting The Most Informative Samples
  8. Unsupervised Training
  9. experiment
  10. Baseline and Dataset
  11. Linear Frequency Cepstral Coefficients
  12. Light Convolution Neural Network
  13. PAF dataset
  14. Training Detail and System Setting
  15. Fails on Cross-domain Dataset
  16. ...and 4 more sections

Figures (4)

  • Figure 1: The visualization of the feature distribution of HAD training dataset(source domain) and ADD2023 test dataset(target domain)
  • Figure 2: The workflow of SDF: On the left is our process of training diverse experts through reverse knowledge distillation. On the right is the process of mining the most informative samples in the target domain using diverse experts. Below is our process for generating labels for these most informative samples, which ultimately participate in the training.
  • Figure 3: The left side shows the average entropy size for all samples of the target dataset with different parameter settings of u, and the right side shows the F1score results for the corresponding u parameter settings when introducing 5000 samples from target dataset.
  • Figure 4: The dissimilarity of predictions among experts