Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Harnessing the Power of Large Vision Language Models for Synthetic Image Detection

Mamadou Keita, Wassim Hamidouche, Hassen Bougueffa, Abdenour Hadid, Abdelmalik Taleb-Ahmed

TL;DR

The paper tackles the challenge of detecting synthetic images produced by diffusion models. It proposes a novel approach that reframes binary classification as an image captioning task using large vision-language models (VLMs), specifically ViTGPT2 and BLIP-2, with LoRA-based fine-tuning for efficiency. By training the VLMs to generate captions that label images as 'real' or 'fake', the method leverages multimodal reasoning to improve generalization to unseen diffusion models. Empirical results on LSUN Bedroom-based diffusion data show that captioning-based VLMs outperform traditional binary detectors, with BLIP-2 exhibiting particularly strong robustness across model subsets. The work highlights the practical potential of multimodal authenticity assessment for applications in security, misinformation mitigation, and content moderation.

Abstract

In recent years, the emergence of models capable of generating images from text has attracted considerable interest, offering the possibility of creating realistic images from text descriptions. Yet these advances have also raised concerns about the potential misuse of these images, including the creation of misleading content such as fake news and propaganda. This study investigates the effectiveness of using advanced vision-language models (VLMs) for synthetic image identification. Specifically, the focus is on tuning state-of-the-art image captioning models for synthetic image detection. By harnessing the robust understanding capabilities of large VLMs, the aim is to distinguish authentic images from synthetic images produced by diffusion-based models. This study contributes to the advancement of synthetic image detection by exploiting the capabilities of visual language models such as BLIP-2 and ViTGPT2. By tailoring image captioning models, we address the challenges associated with the potential misuse of synthetic images in real-world applications. Results described in this paper highlight the promising role of VLMs in the field of synthetic image detection, outperforming conventional image-based detection techniques. Code and models can be found at https://github.com/Mamadou-Keita/VLM-DETECT.

Harnessing the Power of Large Vision Language Models for Synthetic Image Detection

TL;DR

The paper tackles the challenge of detecting synthetic images produced by diffusion models. It proposes a novel approach that reframes binary classification as an image captioning task using large vision-language models (VLMs), specifically ViTGPT2 and BLIP-2, with LoRA-based fine-tuning for efficiency. By training the VLMs to generate captions that label images as 'real' or 'fake', the method leverages multimodal reasoning to improve generalization to unseen diffusion models. Empirical results on LSUN Bedroom-based diffusion data show that captioning-based VLMs outperform traditional binary detectors, with BLIP-2 exhibiting particularly strong robustness across model subsets. The work highlights the practical potential of multimodal authenticity assessment for applications in security, misinformation mitigation, and content moderation.

Abstract

In recent years, the emergence of models capable of generating images from text has attracted considerable interest, offering the possibility of creating realistic images from text descriptions. Yet these advances have also raised concerns about the potential misuse of these images, including the creation of misleading content such as fake news and propaganda. This study investigates the effectiveness of using advanced vision-language models (VLMs) for synthetic image identification. Specifically, the focus is on tuning state-of-the-art image captioning models for synthetic image detection. By harnessing the robust understanding capabilities of large VLMs, the aim is to distinguish authentic images from synthetic images produced by diffusion-based models. This study contributes to the advancement of synthetic image detection by exploiting the capabilities of visual language models such as BLIP-2 and ViTGPT2. By tailoring image captioning models, we address the challenges associated with the potential misuse of synthetic images in real-world applications. Results described in this paper highlight the promising role of VLMs in the field of synthetic image detection, outperforming conventional image-based detection techniques. Code and models can be found at https://github.com/Mamadou-Keita/VLM-DETECT.
Paper Structure (10 sections, 4 equations, 3 figures, 1 table)

This paper contains 10 sections, 4 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Synthetic Image Detection with VLMs
  3. Synthetic Image Detection
  4. Fine-Tuning
  5. ViTGPT2
  6. BLIP-2
  7. Results
  8. Experimental Setup
  9. Results and Analysis
  10. Conclusion

Figures (3)

  • Figure 1: Proposed methodology for synthetic image detection based on the blip-2 architecture.
  • Figure 2: blip-2 fine-tuning with lora.
  • Figure 3: Each sub-figure is a random image from a testing set, labeled below. The 5-digit binary code shows results from ResNet, Xception, deit, ViTGPT2, and blip2 models, where '0' means real and '1' means fake. All generated images are considered fake.