Context and Transcripts Improve Detection of Deepfake Audios of Public Figures

TL;DR

This work tackles the limitation of audio-only deepfake detectors by integrating rich contextual information and transcripts into detection via the Context-based Audio Deepfake Detection (CADD) framework. Using a Journalist-provided Deepfake Dataset (JDD) and a Synthetic Audio Dataset (SYN), along with evaluations on ITW and P$^2$V, the authors demonstrate that context and transcripts substantially boost detection performance across 71 baselines and three datasets. CADD achieves statistically significant gains, improves robustness against adversarial audio perturbations, and reveals domain-specific patterns in political versus entertainment deepfakes. The approach provides a practical, scalable path toward more reliable public-figure deepfake detection with potential extensions to multimedia modalities.

Abstract

Humans use context to assess the veracity of information. However, current audio deepfake detectors only analyze the audio file without considering either context or transcripts. We create and analyze a Journalist-provided Deepfake Dataset (JDD) of 255 public deepfakes which were primarily contributed by over 70 journalists since early 2024. We also generate a synthetic audio dataset (SYN) of dead public figures and propose a novel Context-based Audio Deepfake Detector (CADD) architecture. In addition, we evaluate performance on two large-scale datasets: ITW and P$^2$V. We show that sufficient context and/or the transcript can significantly improve the efficacy of audio deepfake detectors. Performance (measured via F1 score, AUC, and EER) of multiple baseline audio deepfake detectors and traditional classifiers can be improved by 5%-37.58% in F1-score, 3.77%-42.79% in AUC, and 6.17%-47.83% in EER. We additionally show that CADD, via its use of context and/or transcripts, is more robust to 5 adversarial evasion strategies, limiting performance degradation to an average of just -0.71% across all experiments. Code, models, and datasets are available at our project page: https://sites.northwestern.edu/nsail/cadd-context-based-audio-deepfake-detection (access restricted during review).

Figures (19)

• Figure 1: Performance comparison of state-of-the-art (a) and traditional machine learning (b) baselines and our CADD configurations (CADD(T), CADD(C), and CADD(T+C)). Each point represents a model's average performance score computed on our Journalist-provided Deepfake Dataset (JDD), with the same color denoting the same baseline across different configurations.
• Figure 2: Performance Robustness Under Audio Manipulations (JDD dataset) The figure shows the absolute percentage difference between the results (in terms of Avg scores) obtained on the perturbed JDD test set and those obtained on the original, unperturbed test set for the subset of 4 best-performing SOTA baselines and related CADD configurations. SOTA models are separated by black vertical lines, with baseline results highlighted in a dashed rectangle.
• Figure 3: The architecture of our CADD context fusion module has three components: 1) the context encoder, 2) the deepfake audio detection backbone, and 3) the fusion module and classification head.
• Figure S1: Performance comparison of state-of-the-art (a) and traditional machine learning (b) baselines and our CADD configurations (CADD(T), CADD(C), and CADD(T+C)). Each point represents a model's F1 score computed on our Journalist-provided Deepfake Dataset (JDD), with the same color denoting the same baseline across different configurations.
• Figure S2: Performance comparison of state-of-the-art (a) and traditional machine learning (b) baselines and our CADD configurations (CADD(T), CADD(C), and CADD(T+C)). Each point represents a model's AUC score computed on our Journalist-provided Deepfake Dataset (JDD), with the same color denoting the same baseline across different configurations.