Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

Bin Cao, Jianhao Yuan, Yexin Liu, Jian Li, Shuyang Sun, Jing Liu, Bo Zhao

TL;DR

A comprehensive artifact taxonomy is developed and a dataset of synthetic images with artifact annotations is constructed for fine-tuning VLM, named SynArtifact-1K, which exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%.

Abstract

In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.

SynArtifact: Classifying and Alleviating Artifacts in Synthetic Images via Vision-Language Model

TL;DR

A comprehensive artifact taxonomy is developed and a dataset of synthetic images with artifact annotations is constructed for fine-tuning VLM, named SynArtifact-1K, which exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%.

Abstract

In the rapidly evolving area of image synthesis, a serious challenge is the presence of complex artifacts that compromise perceptual realism of synthetic images. To alleviate artifacts and improve quality of synthetic images, we fine-tune Vision-Language Model (VLM) as artifact classifier to automatically identify and classify a wide range of artifacts and provide supervision for further optimizing generative models. Specifically, we develop a comprehensive artifact taxonomy and construct a dataset of synthetic images with artifact annotations for fine-tuning VLM, named SynArtifact-1K. The fine-tuned VLM exhibits superior ability of identifying artifacts and outperforms the baseline by 25.66%. To our knowledge, this is the first time such end-to-end artifact classification task and solution have been proposed. Finally, we leverage the output of VLM as feedback to refine the generative model for alleviating artifacts. Visualization results and user study demonstrate that the quality of images synthesized by the refined diffusion model has been obviously improved.
Paper Structure (29 sections, 5 equations, 13 figures, 2 tables)

This paper contains 29 sections, 5 equations, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Synthetic Image Assessment
  4. Assessment with Vision-Language Model
  5. Reinforcement Learning from AI Feedback
  6. Methodology
  7. Overview
  8. Dataset Construction
  9. Artifact Taxonomy
  10. Image Generation and Annotation
  11. Automatic Artifact Classification via Fine-tuning Vision-Language Model
  12. Artifact Classification as AI Feedback for Alleviating Artifact
  13. Experiments
  14. Dataset & Setting
  15. Artifact Classification Results
  16. ...and 14 more sections

Figures (13)

  • Figure 1: The Proposed Method. The entire pipeline consists of three components: dataset construction (see \ref{['sec:artifacts taxonomy and dataset construction']}), fine-tuning VLM as artifact classifier (see \ref{['sec:Automatic Artifact Detection with Vision-Language Model']}) and artifact alleviation via RLAIF (see \ref{['sec:Detection4RL']}).
  • Figure 2: Artifact Taxonomy. The entire artifact taxonomy contains 13 kinds of artifacts. Each kind of artifact is accompanied with explanation.
  • Figure 3: Our Annotation Paradigm. Each synthetic image is annotated with coordinates, label and caption of artifacts.
  • Figure 4: Distribution of Artifact Annotation. Distortion, omission, and duplication are dominant artifacts while artifacts related to lighting are infrequent.
  • Figure 5: Question-Answer Template for Artifact Classification. The template includes a set of options. Each option is accompanied with a detailed explanation. Reference answers are used to ensure the standardized format for response.
  • ...and 8 more figures