DiffusionFace: Towards a Comprehensive Dataset for Diffusion-Based Face Forgery Analysis

TL;DR

This work introduces DiffusionFace, the first diffusion-based facial forgery dataset, to address the emergence of high-quality forgeries produced by diffusion models. It combines real MM-CelebA-HQ faces with synthetic forgeries generated by 11 diffusion models across unconditional and five conditional categories (Text2Img, Img2Img, Inpaint, DiffSwap), totaling 600k images plus internet-sourced eval data. The authors provide rich metadata, alignment and quality controls, and a comprehensive evaluation protocol covering within-domain, cross-domain, post-processing, cross-data, and in-the-wild scenarios, plus extensive detector analyses and frequency-domain insights. Overall, DiffusionFace offers a substantial, realistic benchmark and baseline results to spur development of robust, diffusion-aware facial forgery detectors with real-world applicability.

Abstract

The rapid progress in deep learning has given rise to hyper-realistic facial forgery methods, leading to concerns related to misinformation and security risks. Existing face forgery datasets have limitations in generating high-quality facial images and addressing the challenges posed by evolving generative techniques. To combat this, we present DiffusionFace, the first diffusion-based face forgery dataset, covering various forgery categories, including unconditional and Text Guide facial image generation, Img2Img, Inpaint, and Diffusion-based facial exchange algorithms. Our DiffusionFace dataset stands out with its extensive collection of 11 diffusion models and the high-quality of the generated images, providing essential metadata and a real-world internet-sourced forgery facial image dataset for evaluation. Additionally, we provide an in-depth analysis of the data and introduce practical evaluation protocols to rigorously assess discriminative models' effectiveness in detecting counterfeit facial images, aiming to enhance security in facial image authentication processes. The dataset is available for download at \url{https://github.com/Rapisurazurite/DiffFace}.

Figures (14)

• Figure 1: Examples of generated images. The first row illustrates Unconditional Image Generation, while the second row showcases Conditional Image Generation.
• Figure 2: Pipeline of our face forgery approches. (a) adopt pretrained diffusion models to generate forgery image directly. (b)-(e) represent conditional image generations conditioned on text prompts, image constraints, context cues, and identity, respectively.
• Figure 3: Illustration of the composition of the DiffusionFace dataset. Our dataset comprises 600,000 images, with 5% consisting of 30,000 images.
• Figure 4: Visualization of Image-Guided Image Generation, Image Inpainting and Diffusion-Based Face Swap. In Img2Img, images are generated with varying $t_0$ parameters, progressively increasing the modification. In DiffInpaint, the diffusion model modifies only the masked area based on the image context. In DiffSwap, only the identity is altered.
• Figure 5: Mean of the DFT spectrum from real and generated images.