Semantic Contextualization of Face Forgery: A New Definition, Dataset, and Detection Method

TL;DR

The paper reframes face forgery through a semantic lens, defining forgery as alterations to semantic face attributes that cross human perceptual thresholds. It introduces the FFSC dataset with a hierarchical label graph linking global attributes to local regions and two testing protocols to probe generalization across unseen manipulations and attributes. A semantics-oriented detection method is proposed, leveraging a probabilistic label model and bi-level optimization (Auto-$\lambda$) to prioritize the primary real-vs-fake task while propagating semantic and regional cues. Experiments demonstrate FFSC’s effectiveness as a challenging testbed and its utility as a training set to improve generalization, with the proposed method outperforming traditional binary or multi-class detectors and reducing reliance on manipulation-specific cues.

Abstract

In recent years, deep learning has greatly streamlined the process of manipulating photographic face images. Aware of the potential dangers, researchers have developed various tools to spot these counterfeits. Yet, none asks the fundamental question: What digital manipulations make a real photographic face image fake, while others do not? In this paper, we put face forgery in a semantic context and define that computational methods that alter semantic face attributes to exceed human discrimination thresholds are sources of face forgery. Following our definition, we construct a large face forgery image dataset, where each image is associated with a set of labels organized in a hierarchical graph. Our dataset enables two new testing protocols to probe the generalizability of face forgery detectors. Moreover, we propose a semantics-oriented face forgery detection method that captures label relations and prioritizes the primary task (i.e., real or fake face detection). We show that the proposed dataset successfully exposes the weaknesses of current detectors as the test set and consistently improves their generalizability as the training set. Additionally, we demonstrate the superiority of our semantics-oriented method over traditional binary and multi-class classification-based detectors.

Figures (9)

• Figure 1: Illustration of diffusion autoencoders Preechakul_2022_CVPR on age manipulation. By varying the age parameter $d_\mathrm{age}$, which controls the movement of the image latent along the age direction, we create a set of age-manipulated images, only a subset of which are considered fake according to our definition (e.g., those with $d_\mathrm{age} \le -0.20$). A more negative number indicates an older age.
• Figure 2: (a) Hierarchical graph for label relation encoding in FFSC. We partition the root node, denoted as face, into five global face attribute nodes, age, expression, gender, identity, and pose, each of which is further connected to a set of leaf nodes, representing local face regions. (b) Parsing of the face in Fig. \ref{['fig: degree_control_fakeness']}(a) into non-overlapping local face regions.
• Figure 3: Face manipulation techniques adopted in FFSC. Each subfigure displays the original and manipulated face images on the left and right, respectively.
• Figure 4: Human discrimination distributions of face attributes and regions. Zoom in for improved visibility.
• Figure 5: Weight dynamics during bi-level optimization.