Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Few-shot Defect Image Generation based on Consistency Modeling

Qingfeng Shi, Jing Wei, Fei Shen, Zhengtao Zhang

TL;DR

The proposed DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation.

Abstract

Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zero-shot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at https://github.com/FFDD-diffusion/DefectDiffu.

Few-shot Defect Image Generation based on Consistency Modeling

TL;DR

The proposed DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation.

Abstract

Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zero-shot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at https://github.com/FFDD-diffusion/DefectDiffu.
Paper Structure (27 sections, 12 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 27 sections, 12 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Controllable Image Generation
  4. Diffusion-based generation.
  5. Text-guided Image Generation.
  6. Defect Image Generation
  7. Method
  8. Preliminaries
  9. Text-guided Disentangled Integrated Architecture
  10. Double-free
  11. Mask Generation
  12. Adaptive Attention-enhanced Loss
  13. Zero-shot Generation
  14. Experiment
  15. Experiment Settings
  16. ...and 12 more sections

Figures (5)

  • Figure 1: Example of intra-product background and inter-product defect consistency
  • Figure 2: Overall framework of DefectDiffu. During training, real images and masks are encoded by Image Encoder into feature spaces, which are then used for training the Generation Module and the Mask Generation Module. The real masks are resized for calculating the Adaptive Loss Ratio. During testing, generated image features and mask features are decoded by Image Decoder to obtain image-mask pairs.
  • Figure 3: Comparison of the generation results.
  • Figure 4: Generation results under different strengths, with defect strength gradually increasing from left to right.
  • Figure 5: Zero-shot generation results. A-B means to transfer source defect A to target product B.