Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synthetic Art Generation and DeepFake Detection A Study on Jamini Roy Inspired Dataset

Kushal Agrawal, Romi Banerjee

TL;DR

This study investigates diffusion-based art generation within the Jamini Roy style and its implications for deepfake detection in non-Western art. By fine-tuning Stable Diffusion 3 with DreamBooth and augmenting generation with ControlNet and IPAdapter, the authors create a Jamini Roy–inspired dataset of real and AI-generated works and conduct thorough qualitative, quantitative, and frequency-domain analyses. They evaluate three state-of-the-art detectors, finding that diffusion-induced artifacts can be obscured by advanced generation techniques, though residual frequency-domain cues persist. The work highlights challenges in detecting culturally specific AI-generated art and lays groundwork for hybrid detection approaches and broader cross-cultural forensic research.

Abstract

The intersection of generative AI and art is a fascinating area that brings both exciting opportunities and significant challenges, especially when it comes to identifying synthetic artworks. This study takes a unique approach by examining diffusion-based generative models in the context of Indian art, specifically focusing on the distinctive style of Jamini Roy. To explore this, we fine-tuned Stable Diffusion 3 and used techniques like ControlNet and IPAdapter to generate realistic images. This allowed us to create a new dataset that includes both real and AI-generated artworks, which is essential for a detailed analysis of what these models can produce. We employed various qualitative and quantitative methods, such as Fourier domain assessments and autocorrelation metrics, to uncover subtle differences between synthetic images and authentic pieces. A key takeaway from recent research is that existing methods for detecting deepfakes face considerable challenges, especially when the deepfakes are of high quality and tailored to specific cultural contexts. This highlights a critical gap in current detection technologies, particularly in light of the challenges identified above, where high-quality and culturally specific deepfakes are difficult to detect. This work not only sheds light on the increasing complexity of generative models but also sets a crucial foundation for future research aimed at effective detection of synthetic art.

Synthetic Art Generation and DeepFake Detection A Study on Jamini Roy Inspired Dataset

TL;DR

This study investigates diffusion-based art generation within the Jamini Roy style and its implications for deepfake detection in non-Western art. By fine-tuning Stable Diffusion 3 with DreamBooth and augmenting generation with ControlNet and IPAdapter, the authors create a Jamini Roy–inspired dataset of real and AI-generated works and conduct thorough qualitative, quantitative, and frequency-domain analyses. They evaluate three state-of-the-art detectors, finding that diffusion-induced artifacts can be obscured by advanced generation techniques, though residual frequency-domain cues persist. The work highlights challenges in detecting culturally specific AI-generated art and lays groundwork for hybrid detection approaches and broader cross-cultural forensic research.

Abstract

The intersection of generative AI and art is a fascinating area that brings both exciting opportunities and significant challenges, especially when it comes to identifying synthetic artworks. This study takes a unique approach by examining diffusion-based generative models in the context of Indian art, specifically focusing on the distinctive style of Jamini Roy. To explore this, we fine-tuned Stable Diffusion 3 and used techniques like ControlNet and IPAdapter to generate realistic images. This allowed us to create a new dataset that includes both real and AI-generated artworks, which is essential for a detailed analysis of what these models can produce. We employed various qualitative and quantitative methods, such as Fourier domain assessments and autocorrelation metrics, to uncover subtle differences between synthetic images and authentic pieces. A key takeaway from recent research is that existing methods for detecting deepfakes face considerable challenges, especially when the deepfakes are of high quality and tailored to specific cultural contexts. This highlights a critical gap in current detection technologies, particularly in light of the challenges identified above, where high-quality and culturally specific deepfakes are difficult to detect. This work not only sheds light on the increasing complexity of generative models but also sets a crucial foundation for future research aimed at effective detection of synthetic art.

Paper Structure

This paper contains 23 sections, 6 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction and Background
  2. Literature survey
  3. Image-based Deepfake Detection
  4. Frequency-based Deepfake Detection
  5. Dataset Creation and Novelty
  6. Step 1: Image Extraction
  7. Step 2: Selection of Hard Samples
  8. Step 3: Fine-tuning Stable Diffusion
  9. Step 4: Image Generation
  10. Stable Diffusion 3
  11. Stable Diffusion 3 with ControlNet and IPAdapter
  12. Analysis
  13. Qualitative Analysis
  14. Quantitative Analysis
  15. Frequency Domain Analysis
  16. ...and 8 more sections

Figures (6)

  • Figure 1: The directed graphical model
  • Figure 3: Comparison of Metrics: The SSIM, PSNR, and Histogram Correlation distributions are shown for images generated with and without ControlNet and IPAdapter
  • Figure 4: Power Spectrum And Phase Spectrum of the Original Artwork
  • Figure 5: Power Spectrum And Phase Spectrum of Stable Diffusion Generated Image without ControlNet and IPAdapter
  • Figure 6: Power Spectrum And Phase Spectrum of Stable Diffusion Generated Image with ControlNet and IPAdapter
  • ...and 1 more figures