ViGText: Deepfake Image Detection with Vision-Language Model Explanations and Graph Neural Networks

TL;DR

ViGText addresses the challenge of detecting deepfakes that generalize poorly to user-customized models and resist adversarial perturbations. It introduces a dual-graph framework that fuses image patches (with spatial and DCT frequency features) and VLLM-generated explanations, enabling robust cross-modal reasoning via Graph Neural Networks. The approach yields state-of-the-art detection accuracy, strong generalization to fine-tuned Stable Diffusion variants, and notable robustness against foundation-model-based attacks, while maintaining competitive runtime. This work advances practical deepfake defense by leveraging contextual explanations within a graph-based integration of visual and textual signals, setting a new standard for media authenticity.

Abstract

The rapid rise of deepfake technology, which produces realistic but fraudulent digital content, threatens the authenticity of media. Traditional deepfake detection approaches often struggle with sophisticated, customized deepfakes, especially in terms of generalization and robustness against malicious attacks. This paper introduces ViGText, a novel approach that integrates images with Vision Large Language Model (VLLM) Text explanations within a Graph-based framework to improve deepfake detection. The novelty of ViGText lies in its integration of detailed explanations with visual data, as it provides a more context-aware analysis than captions, which often lack specificity and fail to reveal subtle inconsistencies. ViGText systematically divides images into patches, constructs image and text graphs, and integrates them for analysis using Graph Neural Networks (GNNs) to identify deepfakes. Through the use of multi-level feature extraction across spatial and frequency domains, ViGText captures details that enhance its robustness and accuracy to detect sophisticated deepfakes. Extensive experiments demonstrate that ViGText significantly enhances generalization and achieves a notable performance boost when it detects user-customized deepfakes. Specifically, average F1 scores rise from 72.45% to 98.32% under generalization evaluation, and reflects the model's superior ability to generalize to unseen, fine-tuned variations of stable diffusion models. As for robustness, ViGText achieves an increase of 11.1% in recall compared to other deepfake detection approaches. When facing targeted attacks that exploit its graph-based architecture, ViGText limits classification performance degradation to less than 4%. ViGText uses detailed visual and textual analysis to set a new standard for detecting deepfakes, helping ensure media authenticity and information integrity.

Figures (12)

• Figure 1: An overview of the proposed ViGText's approach.
• Figure 2: An Illustration of the threat model.
• Figure 3: Block diagram of the proposed ViGText pipeline, illustrating the key components and processes.
• Figure 4: Generated image misclassified as real by DE-FAKE sha2023fake.
• Figure 5: Image overlay with grid: (a) original, (b) with grid overlay.