On the Holistic Approach for Detecting Human Image Forgery

TL;DR

This work tackles the challenge of detecting human image forgeries that range from localized facial manipulations to fully synthesized full-body images. It introduces HuForDet, a dual-branch detector with a face forgery branch employing heterogeneous RGB and adaptively scaled frequency (adaLoG) experts, and a contextualized branch that uses a Multi-Modal Large Language Model (MLLM) to reason about global semantic consistency, guided by a self-assessed confidence token. A confidence-aware fusion network combines the two branches, enabling robust detection across forgery types, and the HuFor dataset unifies face-forensics data with a new corpus of full-body synthetic humans. Empirical results show state-of-the-art performance on HuFor and strong generalization across datasets, with detailed ablations confirming the benefits of the adaptive LoG and the confidence-guided fusion. This approach advances practical defense against evolving AI-generated human forgeries by integrating local artifact analysis with global semantic validation.

Abstract

The rapid advancement of AI-generated content (AIGC) has escalated the threat of deepfakes, from facial manipulations to the synthesis of entire photorealistic human bodies. However, existing detection methods remain fragmented, specializing either in facial-region forgeries or full-body synthetic images, and consequently fail to generalize across the full spectrum of human image manipulations. We introduce HuForDet, a holistic framework for human image forgery detection, which features a dual-branch architecture comprising: (1) a face forgery detection branch that employs heterogeneous experts operating in both RGB and frequency domains, including an adaptive Laplacian-of-Gaussian (LoG) module designed to capture artifacts ranging from fine-grained blending boundaries to coarse-scale texture irregularities; and (2) a contextualized forgery detection branch that leverages a Multi-Modal Large Language Model (MLLM) to analyze full-body semantic consistency, enhanced with a confidence estimation mechanism that dynamically weights its contribution during feature fusion. We curate a human image forgery (HuFor) dataset that unifies existing face forgery data with a new corpus of full-body synthetic humans. Extensive experiments show that our HuForDet achieves state-of-the-art forgery detection performance and superior robustness across diverse human image forgeries.

Figures (5)

• Figure 1: (a) Beyond facial forgeries, AIGC methods enable the synthesis of full-body human images, introducing distinctive anatomical anomalies such as an additional finger, unnaturally smooth skin, and three-legged artifacts. (b) The Laplacian of Gaussian (LoG) operator is an effective blob detector for identifying regions with rapid intensity changes, which often correspond to facial forgery artifacts. However, conventional LoG-based detectors hifi_net_xiaoguomasi2020two rely on a fixed scale parameter $\sigma$, capturing only a narrow subset of these artifact patterns. Colored overlays show LoG blob detections at different scales $\sigma$: yellow ($\sigma{=}1$) and red ($\sigma{=}5$) in the first row highlight unnaturally bright mouth regions and blending artifacts; green ($\sigma{=}9$) and blue ($\sigma{=}13$) in the second row emphasize abnormal skin textures. Our adaptive LoG (Sec. \ref{['subsec:gated_svle']}) overcomes this limitation by learning optimal scales, adaptive to different spatial locations. (c) Our proposed HuForDet (Fig. \ref{['fig_archi']}) achieves state-of-the-art performance on detecting both partial-manipulation (e.g., face-swap) and fully synthesized forgeries (e.g., GAN-generated faces, diffusion-generated full-body images) on our proposed HuFor dataset (Sec. \ref{['sec:dataset']}).
• Figure 2: Our HuForDet comprises two branches: a face forgery detection branch ($\mathcal{F}_{\text{face}}$) and a contextualized forgery detection branch ($\mathcal{F}_{\text{ctx}}$), which are introduced in Sec. \ref{['sec:face_branch']} and Sec. \ref{['subsec:context_branch']}, respectively. Specifically, $\mathcal{F}_{\text{face}}$ analyzes cropped face regions $\mathbf{I}_{\text{face}}$ using heterogeneous RGB spatial (i.e., $E_1$ and $E_2$) and frequency domain (i.e., $E_3$ and $E_4$) experts, and then it generates a facial forgery representation $\mathbf f_{\text{face}}\in\mathbb R^{d}$. Also, $\mathcal{F}_{\text{ctx}}$ processes the input image $\mathbf{I}$ to produce a contexualized forgery representation $\mathbf f_{\text{ctx}}\in\mathbb R^{d}$ and a self-assessed confidence $c\in[0,1]$. A confidence-aware fusion module $\mathcal{G}$ aggregates $\mathbf f_{\text{face}}$ and $\mathbf f_{\text{ctx}}$ to produce a holistic representation for the final forgery prediction.
• Figure 3: (a) Examples of celebrity images generated by different diffusion personalized models. (b) Given the image, we use the Gemini-$2.0$ Pro to produce corresponding text annotations.
• Figure 4: Analysis of (a) gate scores and (b) confidence scores across six forgery categories, which are defined in liu2025benchmarking for digital generation and manipulation.
• Figure 5: (a) Visualizations on face regions and feature maps obtained via adaLoG blocks from $E_3$ and $E_4$, respectively. (b) The $\mathcal{F}_{\text{ctx}}$ focuses on forged regions such as facial manipulations in the first example and anomalous finger artifacts in the second. (c) Face swap detection performance.