Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deepfake: Definitions, Performance Metrics and Standards, Datasets and Benchmarks, and a Meta-Review

Enes Altuncu, Virginia N. L. Franqueira, Shujun Li

TL;DR

This paper gives a comprehensive overview of deepfake, covering multiple important aspects of this emerging concept, including (1) different definitions, (2) commonly used performance metrics and standards, and (3) deepfake-related datasets.

Abstract

Recent advancements in AI, especially deep learning, have contributed to a significant increase in the creation of new realistic-looking synthetic media (video, image, and audio) and manipulation of existing media, which has led to the creation of the new term ``deepfake''. Based on both the research literature and resources in English and in Chinese, this paper gives a comprehensive overview of deepfake, covering multiple important aspects of this emerging concept, including 1) different definitions, 2) commonly used performance metrics and standards, and 3) deepfake-related datasets, challenges, competitions and benchmarks. In addition, the paper also reports a meta-review of 12 selected deepfake-related survey papers published in 2020 and 2021, focusing not only on the mentioned aspects, but also on the analysis of key challenges and recommendations. We believe that this paper is the most comprehensive review of deepfake in terms of aspects covered, and the first one covering both the English and Chinese literature and sources.

Deepfake: Definitions, Performance Metrics and Standards, Datasets and Benchmarks, and a Meta-Review

TL;DR

This paper gives a comprehensive overview of deepfake, covering multiple important aspects of this emerging concept, including (1) different definitions, (2) commonly used performance metrics and standards, and (3) deepfake-related datasets.

Abstract

Recent advancements in AI, especially deep learning, have contributed to a significant increase in the creation of new realistic-looking synthetic media (video, image, and audio) and manipulation of existing media, which has led to the creation of the new term ``deepfake''. Based on both the research literature and resources in English and in Chinese, this paper gives a comprehensive overview of deepfake, covering multiple important aspects of this emerging concept, including 1) different definitions, 2) commonly used performance metrics and standards, and 3) deepfake-related datasets, challenges, competitions and benchmarks. In addition, the paper also reports a meta-review of 12 selected deepfake-related survey papers published in 2020 and 2021, focusing not only on the mentioned aspects, but also on the analysis of key challenges and recommendations. We believe that this paper is the most comprehensive review of deepfake in terms of aspects covered, and the first one covering both the English and Chinese literature and sources.
Paper Structure (38 sections, 22 equations, 7 figures, 3 tables)

This paper contains 38 sections, 22 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Definitions of the Term Deepfake
  3. Scope and Contribution
  4. Methodology
  5. Deepfake-Related Performance Metrics & Standards
  6. The Confusion Matrix
  7. Precision and Recall
  8. True and False Positive Rates
  9. True and False Negative Rates
  10. Sensitivity and Specificity
  11. Equal Error Rate
  12. Accuracy and F-Score
  13. Receiver Operating Characteristic Curve and Area Under Curve
  14. Log Loss
  15. Extension to Multi-class Classifiers
  16. ...and 23 more sections

Figures (7)

  • Figure 1: A representative ROC curve showing how TPR and FPR change w.r.t. the (hidden) threshold $t$. The area under the (ROC) curve (AUC) is shown in grey.
  • Figure 2: Screenshot of leaderboard with top five finalists of the DFDC competition.
  • Figure 3: Screenshot of leaderboard with top three finalists of the Face Anti-spoofing Challenge 2021 competition.
  • Figure 4: Illustration of the FaceForensics Benchmark in terms of submission and result.
  • Figure 5: Final results for the 2020 CelebA-Spoof Face Anti-Spoofing Challenge.
  • ...and 2 more figures