Bi-LORA: A Vision-Language Approach for Synthetic Image Detection

TL;DR

Bi-LORA introduces a vision-language model–driven framework for synthetic image detection by reframing the binary real-vs-fake task as image captioning. Through LoRA-based fine-tuning of a BLIP2-like VLm, it achieves robust, zero-shot generalization to unseen diffusion- and GAN-generated images while requiring only a tiny fraction of trainable parameters. Extensive experiments across diffusion and GAN generators, degradation scenarios, and diverse datasets demonstrate strong cross-generator performance and competitive advantages over state-of-the-art detectors. The work highlights the complementarity of vision-language representations for forensic detection and points to future directions in multitask attribution and knowledge distillation for scalable deployment.

Abstract

Advancements in deep image synthesis techniques, such as generative adversarial networks (GANs) and diffusion models (DMs), have ushered in an era of generating highly realistic images. While this technological progress has captured significant interest, it has also raised concerns about the potential difficulty in distinguishing real images from their synthetic counterparts. This paper takes inspiration from the potent convergence capabilities between vision and language, coupled with the zero-shot nature of vision-language models (VLMs). We introduce an innovative method called Bi-LORA that leverages VLMs, combined with low-rank adaptation (LORA) tuning techniques, to enhance the precision of synthetic image detection for unseen model-generated images. The pivotal conceptual shift in our methodology revolves around reframing binary classification as an image captioning task, leveraging the distinctive capabilities of cutting-edge VLM, notably bootstrapping language image pre-training (BLIP2). Rigorous and comprehensive experiments are conducted to validate the effectiveness of our proposed approach, particularly in detecting unseen diffusion-generated images from unknown diffusion-based generative models during training, showcasing robustness to noise, and demonstrating generalization capabilities to GANs. The obtained results showcase an impressive average accuracy of 93.41% in synthetic image detection on unseen generation models. The code and models associated with this research can be publicly accessed at https://github.com/Mamadou-Keita/VLM-DETECT.

Figures (4)

• Figure 1: Examples of real sample and synthetic images generated by various generators in our experiments. From left to righ, Top row: Real yu2015lsun, ADM dhariwal2021diffusion, ldm. Middle row: DDPM ho2020denoising, IDDPM nichol2021improved, PNDM liu2022pseudo. Bottom row: Stable Diffusion v1.4 rombach2022high, GLIDE nichol2021glide, DALL·E 3 betker2023improving.
• Figure 2: Bi-LORA finetuning for synthetic image detector.
• Figure 3: Bi-LORA fine-tuning with lora.
• Figure 4: Each sub-figure represents a random compressed (JPEG q=65) image from a testing set, with predicted labels provided below. The 5-digit binary code shows results from ResNet, Xception, deit, ViTGPT2, and Bi-LORA models, where '0' means real and '1' means fake. It is important to note that all generated images are considered fake.