PhytoSynth: Leveraging Multi-modal Generative Models for Crop Disease Data Generation with Novel Benchmarking and Prompt Engineering Approach

TL;DR

The paper tackles the challenge of assembling large crop-disease image datasets by proposing a multi-modal text-to-image diffusion pipeline based on Stable Diffusion variants (SDXL, SD3.5M, SD3.5L) and augmenting them with Dreambooth and LoRA fine-tuning. It introduces a novel computational benchmarking framework for agricultural generative models, evaluating memory, power, energy, and perceptual quality (LPIPS) while generating 500 images from a small field-sample set in about 1.5 hours. SD3.5M emerges as the most efficient variant, achieving competitive perceptual similarity (LPIPS ~0.34) with markedly lower resource usage than SDXL and SD3.5L. The study also emphasizes careful dataset construction, prompt engineering, and the need for domain-specific vocabulary in prompts, offering practical guidance for resource-constrained researchers and establishing benchmarks to compare future agricultural GMs.

Abstract

Collecting large-scale crop disease images in the field is labor-intensive and time-consuming. Generative models (GMs) offer an alternative by creating synthetic samples that resemble real-world images. However, existing research primarily relies on Generative Adversarial Networks (GANs)-based image-to-image translation and lack a comprehensive analysis of computational requirements in agriculture. Therefore, this research explores a multi-modal text-to-image approach for generating synthetic crop disease images and is the first to provide computational benchmarking in this context. We trained three Stable Diffusion (SD) variants-SDXL, SD3.5M (medium), and SD3.5L (large)-and fine-tuned them using Dreambooth and Low-Rank Adaptation (LoRA) fine-tuning techniques to enhance generalization. SD3.5M outperformed the others, with an average memory usage of 18 GB, power consumption of 180 W, and total energy use of 1.02 kWh/500 images (0.002 kWh per image) during inference task. Our results demonstrate SD3.5M's ability to generate 500 synthetic images from just 36 in-field samples in 1.5 hours. We recommend SD3.5M for efficient crop disease data generation.

Figures (6)

• Figure 1: Overview of proposed dataset for this research study: (a) our dataset that consists of variations, such as shadow, plastic mulch, fruit, soil background, and a hand, and (b) open-access dataset that imparts additional diversity in terms of curled leaves, focused symptoms, blurred and colored backgrounds.
• Figure 2: Pipeline for generative model training and synthetic image generation. Left box: The workflow begins with data acquisition, incorporating both captured and open-access images. Right box: The acquired images undergo dataset filtering, resizing, and class sorting to train a generative model using Dreambooth and LoRA fine-tuning approaches. Prompt engineering is employed to generate user-centered desired number of synthetic images.
• Figure 3: Schematic representation of the generative pipeline for disease image synthesis using a stable diffusion model: (a) In the pixel space, input images are encoded and generated images are decoded after training, (b) In the latent space, the model undergoes forward (noising process) and reverse (denoising process) diffusion process, and (c) The conditioning stage enables text-to-image generation using descriptive prompts while ensuring relevant disease features are preserved.
• Figure 4: Comparison of memory usage and power consumption for SDXL, SD3.5M, and SD3.5L models during training and inference stages. The six graphs illustrate variations in resource utilization, with memory usage and power consumption plotted on the y-axes for training and inferencing stages. Note: The above graphs report metrics based on generating 500 image samples.
• Figure 5: Time taken (hours) vs. LPIPS score measured for all the three variants of a multi-modal generative models.