In Anticipation of Perfect Deepfake: Identity-anchored Artifact-agnostic Detection under Rebalanced Deepfake Detection Protocol

TL;DR

The paper tackles the vulnerability of artifact-based deepfake detectors to near-perfect forgeries by introducing the Rebalanced Deepfake Detection Protocol (RDDP), which balances artifacts across genuine and forged samples using white-hat reconstructions or surrogate perturbations. It then presents ID-Miner, an identity-anchored, artifact-agnostic detector that emphasizes action sequences over appearance by learning frame-level FAU-based embeddings with an artifact-agnostic loss and aggregating them into a video-level representation with an identity-anchored loss. Across two major portrait-video benchmarks (VoxCeleb for FR and Celeb-DF for FS), ID-Miner consistently outperforms 12 baselines under both conventional and RDDP evaluations, and shows strong generalization and pup-reid capabilities. The work highlights the necessity of evaluating and designing detectors that rely on robust identity-action cues rather than superficial artifacts, offering a practical path toward identifying perpetrators of deepfake media while acknowledging ethical considerations and limitations.

Abstract

As deep generative models advance, we anticipate deepfakes achieving "perfection"-generating no discernible artifacts or noise. However, current deepfake detectors, intentionally or inadvertently, rely on such artifacts for detection, as they are exclusive to deepfakes and absent in genuine examples. To bridge this gap, we introduce the Rebalanced Deepfake Detection Protocol (RDDP) to stress-test detectors under balanced scenarios where genuine and forged examples bear similar artifacts. We offer two RDDP variants: RDDP-WHITEHAT uses white-hat deepfake algorithms to create 'self-deepfakes,' genuine portrait videos with the resemblance of the underlying identity, yet carry similar artifacts to deepfake videos; RDDP-SURROGATE employs surrogate functions (e.g., Gaussian noise) to process both genuine and forged examples, introducing equivalent noise, thereby sidestepping the need of deepfake algorithms. Towards detecting perfect deepfake videos that aligns with genuine ones, we present ID-Miner, a detector that identifies the puppeteer behind the disguise by focusing on motion over artifacts or appearances. As an identity-based detector, it authenticates videos by comparing them with reference footage. Equipped with the artifact-agnostic loss at frame-level and the identity-anchored loss at video-level, ID-Miner effectively singles out identity signals amidst distracting variations. Extensive experiments comparing ID-Miner with 12 baseline detectors under both conventional and RDDP evaluations with two deepfake datasets, along with additional qualitative studies, affirm the superiority of our method and the necessity for detectors designed to counter perfect deepfakes.

Figures (10)

• Figure 1: Comparison of the evaluation protocols. The conventional protocol directly contrasts forged and genuine data, allowing deepfake detectors to exploit the distribution shift between the forged and the genuine examples. In contrast, RDDP-whitehat and RDDP-surrogate reduce this bias by 1) processing genuine examples through a white-hat deepfake function ($\mathcal{F}$) to generate reconstructed examples (recon.), and 2) applying a surrogate function ($\mathcal{A}$) to both forged and genuine videos, respectively. By removing the obvious disparities between genuine and deepfake videos, RDDP compels detectors to seek more robust detection cues, e.g., action sequences, as the above frame samples show that the differences in background and sharpness between forged and genuine in conventional is reduced in RDDP-whitehat by changing the genuine and in RDDP-surrogate by transforming both sides uniformly.
• Figure 2: The architecture and training of ID-Miner. ID-Miner employs a hierarchical structure to extract embeddings with frame-level encoders from Facial Action Unit (FAU) attributes, then process the embeddings into a video-level representation. During training, the identity-anchored loss discriminates between video representations of the same individual's action sequences ($\mathbf{a}$ and $\mathbf{a'}$) and those of a different individual ($\mathbf{b}$), irrespective of the facial appearances ($x$, $y$, or $z$). Concurrently, the frame-level encoder is trained by the artifact-agnostic loss to sample frames from videos pre- and post-deepfake transformation to prioritize encoding facial expressions and actions over artifacts.
• Figure 3: Frame distributions. Green $\mathbf{+}$, blue $\circ$, red $\times$ represents frame sampled from $D_{gen}$, $D_{recon}$, $D_{forged}$ for RDDP-whitehat and $D_{gen}$, $\mathcal{A}\circ D_{gen}$, $\mathcal{A}\circ D_{forged}$ for RDDP-surrogate (Resize and GB). The mixture of blue $\circ$ and red $\times$ being separate from the green $\mathbf{+}$ cluster in each plot demonstrates RDDP reducing the distribution shift between genuine and forged examples.
• Figure 4: FAU attribute and embedding distributions. Green $\mathbf{+}$ and red $\times$ represents or embeddings extracted from genuine and forged examples, respectively. The contrast between clear separation (left) and mixture result (right) highlights the effectiveness of frame-level encoder in ID-Miner to be (deepfake) artifact-agnostic.
• Figure 5: Sensitivity tests. We vary the noise intensity level ranging from $0$ to $5$ for the surrogate functions under RDDP-surrogate. ID-Miner (red) delivers comparable performance to the baseline methods under $0$ noise level while exhibiting the least degradation as noise levels increases.

Theorems & Definitions (3)

• Definition 1: conventional protocol
• Definition 2: RDDP-whitehat
• Definition 3: RDDP-surrogate