Scaling Laws for Deepfake Detection

TL;DR

This work builds ScaleDF, the largest-scale deepfake detection dataset to date, to systematically study how model performance scales with data diversity and quantity. It reveals that detection error follows a power-law decay with the number of real domains and the number of deepfake methods, and a double-saturating law with training image count, all under a ViT backbone. The study also examines pre-training, data augmentation, and cross-benchmark generalization, showing that diversity and perturbations improve robustness and cross-benchmark transfer, while scaling alone cannot guarantee robustness to novel forgery types. The findings provide a data-centric blueprint for developing more robust deepfake detectors and establish ScaleDF as a benchmark to guide future scaling-law research in this domain.

Abstract

This paper presents a systematic study of scaling laws for the deepfake detection task. Specifically, we analyze the model performance against the number of real image domains, deepfake generation methods, and training images. Since no existing dataset meets the scale requirements for this research, we construct ScaleDF, the largest dataset to date in this field, which contains over 5.8 million real images from 51 different datasets (domains) and more than 8.8 million fake images generated by 102 deepfake methods. Using ScaleDF, we observe power-law scaling similar to that shown in large language models (LLMs). Specifically, the average detection error follows a predictable power-law decay as either the number of real domains or the number of deepfake methods increases. This key observation not only allows us to forecast the number of additional real domains or deepfake methods required to reach a target performance, but also inspires us to counter the evolving deepfake technology in a data-centric manner. Beyond this, we examine the role of pre-training and data augmentations in deepfake detection under scaling, as well as the limitations of scaling itself.

Figures (7)

• Figure 1: Illustration of ScaleDF, which is the largest deepfake detection dataset across four dimensions: number of real domains, number of deepfake methods, number of videos, and number of images. To ensure a fair comparison, for datasets that do not explicitly provide the number of images, we estimate image counts by sampling one frame per second from the videos. Using this dataset, we present a systematic study on scaling laws for deepfake detection and reveal several insights.
• Figure 2: Statistics of heights and widths of all real faces in the ScaleDF dataset.
• Figure 3: Left: observed power-law scaling as the number of training real domains changes; Right: observed power-law scaling as the number of training deepfake methods changes. $\mu$ represents the mean computed over $10$ repetitions and $7$ test datasets, while $\sigma$ denotes the variance across the $10$ repetitions.
• Figure 4: Left: observed double-saturating power-law scaling as the number of training images changes; $\mu$ represents the mean computed over $5$ repetitions and $7$ test datasets, while $\sigma$ denotes the variance across the $5$ repetitions. Right: performance changes with respect to model sizes.
• Figure 5: Left Top: observed power-law scaling as the number of training real domains changes; Right Top: observed power-law scaling as the number of training deepfake methods changes; Left Bottom: observed double-saturating power-law scaling as the number of training images changes; Right Bottom: performance changes with respect to model sizes. $\mu$ represents the mean computed over repetitions and test datasets, while $\sigma$ denotes the variance across repetitions.