SoK: Systematization and Benchmarking of Deepfake Detectors in a Unified Framework

TL;DR

The paper tackles the fragmentation in deepfake detector evaluation by introducing a unified Systematization and Benchmarking framework (SoK) that organizes detectors into a four-group taxonomy and 13 subgroups guided by 18 Influential Factors. It then systematically evaluates 16 open-source detectors using a novel white-box dataset and three evaluation settings (black-box, gray-box, white-box), highlighting the limited generalizability of detectors in real-world scenarios. Key contributions include the Conceptual Framework, a public evaluation pipeline, and a comprehensive cross-setting assessment that reveals strong performance for spatiotemporal and attention-based architectures but persistent weaknesses in unexplored synthesis detection and cross-dataset generalization. The findings emphasize the need for more generalized, multimodal, and proactive defenses, and offer seven forward-looking directions to strengthen deepfake detection in practice.

Abstract

Deepfakes have rapidly emerged as a serious threat to society due to their ease of creation and dissemination, triggering the accelerated development of detection technologies. However, many existing detectors rely on labgenerated datasets for validation, which may not prepare them for novel, real-world deepfakes. This paper extensively reviews and analyzes state-of-the-art deepfake detectors, evaluating them against several critical criteria. These criteria categorize detectors into 4 high-level groups and 13 finegrained sub-groups, aligned with a unified conceptual framework we propose. This classification offers practical insights into the factors affecting detector efficacy. We evaluate the generalizability of 16 leading detectors across comprehensive attack scenarios, including black-box, white-box, and graybox settings. Our systematized analysis and experiments provide a deeper understanding of deepfake detectors and their generalizability, paving the way for future research and the development of more proactive defenses against deepfakes.

Figures (6)

• Figure 1: Publications per year for deepfake-related keywords.
• Figure 2: Our Five-Step Conceptual Framework: All detection methods adhere to this framework: Step #1 (Deepfake Type), #2 (Detection Methodology), #3 (Data & Preprocessing), #4 (Model & Training), and #5 (Model Validation). From these primary stages, we identify 18 Influential Factors (illustrated as colored capsules) detailed in Sec. \ref{['sub:conceptual-framework']}.
• Figure 3: Gray-box results. Performance (AUC%) of selected deepfake detectors on CelebDF and DFDC datasets. The overall performance of detectors on the DFDC dataset tends to be lower than CelebDF. To ensure a fair cross-evaluation comparison, we exclude CCViT since this model was trained directly on DFDC rather than the standard deepfake dataset, FF++.
• Figure 4: White-box results (Stabilized Set). F1 scores (dashed) and AUC scores (solid) of selected deepfake detectors.
• Figure 5: Black-box results (in-the-wild).