Deep Learning for Deepfakes Creation and Detection: A Survey

TL;DR

This survey surveys deepfake creation and detection, detailing DL-based generation methods (autoencoders, GANs, and StyleGAN) and a broad spectrum of detection approaches. It categorizes image- and video-based detection into handcrafted versus learned features, and further into temporal versus visual-artifact cues, reviewing key datasets and benchmarks. The authors highlight methodological trends, dataset fragmentation, and the need for robust, generalizable detectors, plus directions like platform integration, watermarking, and blockchain-assisted provenance. They also emphasize the importance of explainable AI to support forensic conviction and policy. Overall, the work provides a comprehensive, systems-level view of the evolving deepfake landscape and detection landscape.

Abstract

Deep learning has been successfully applied to solve various complex problems ranging from big data analytics to computer vision and human-level control. Deep learning advances however have also been employed to create software that can cause threats to privacy, democracy and national security. One of those deep learning-powered applications recently emerged is deepfake. Deepfake algorithms can create fake images and videos that humans cannot distinguish them from authentic ones. The proposal of technologies that can automatically detect and assess the integrity of digital visual media is therefore indispensable. This paper presents a survey of algorithms used to create deepfakes and, more importantly, methods proposed to detect deepfakes in the literature to date. We present extensive discussions on challenges, research trends and directions related to deepfake technologies. By reviewing the background of deepfakes and state-of-the-art deepfake detection methods, this study provides a comprehensive overview of deepfake techniques and facilitates the development of new and more robust methods to deal with the increasingly challenging deepfakes.

Figures (8)

• Figure 1: Number of papers related to deepfakes in years from 2016 to 2021, obtained from https://app.dimensions.ai at the end of 2021 with the search keyword "deepfake" applied to full text of scholarly papers.
• Figure 2: Categories of reviewed papers relevant to deepfake detection methods where we divide papers into two major groups, i.e., fake image detection and face video detection.
• Figure 3: A deepfake creation model using two encoder-decoder pairs. Two networks use the same encoder but different decoders for training process (top). An image of face A is encoded with the common encoder and decoded with decoder B to create a deepfake (bottom). The reconstructed image (in the bottom) is the face B with the mouth shape of face A. Face B originally has the mouth of an upside-down heart while the reconstructed face B has the mouth of a conventional heart.
• Figure 4: The GAN architecture consisting of a generator and a discriminator, and each can be implemented by a neural network. The entire system can be trained with backpropagation that allows both networks to improve their capabilities.
• Figure 5: Examples of mixing styles using StyleGAN: the output images are generated by copying a specified subset of styles from source B and taking the rest from source A. a) Copying coarse styles from source B will generate images that have high-level aspects from source B and all colors and finer facial features from source A; b) if copying the styles of middle resolutions from B, the output images will have smaller scale facial features from B and preserve the pose, general face shape, and eyeglasses from A; c) if copying the fine styles from source B, the generated images will have the color scheme and microstructure of source B Karras2019.