Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis

Jaehyun Park, Minyoung Ahn, Minkyu Kim, Jonghyun Lee, Jae-Gil Lee, Dongmin Park

TL;DR

This paper proposes ArtiAgent, which efficiently creates pairs of real and artifact-injected images that synthesize 100K images with rich artifact annotations and demonstrates both efficacy and versatility across diverse applications.

Abstract

Despite recent advances in diffusion models, AI generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, which makes artifact mitigation a highly crucial area of study. Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale, underscoring the need for an automated approach to reliably acquire artifact-annotated datasets. In this paper, we propose ArtiAgent, which efficiently creates pairs of real and artifact-injected images. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images, a synthesis agent that introduces artifacts via artifact injection tools through novel patch-wise embedding manipulation within a diffusion transformer, and a curation agent that filters the synthesized artifacts and generates both local and global explanations for each instance. Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications. Code is available at link.

See and Fix the Flaws: Enabling VLMs and Diffusion Models to Comprehend Visual Artifacts via Agentic Data Synthesis

TL;DR

This paper proposes ArtiAgent, which efficiently creates pairs of real and artifact-injected images that synthesize 100K images with rich artifact annotations and demonstrates both efficacy and versatility across diverse applications.

Abstract

Despite recent advances in diffusion models, AI generated images still often contain visual artifacts that compromise realism. Although more thorough pre-training and bigger models might reduce artifacts, there is no assurance that they can be completely eliminated, which makes artifact mitigation a highly crucial area of study. Previous artifact-aware methodologies depend on human-labeled artifact datasets, which are costly and difficult to scale, underscoring the need for an automated approach to reliably acquire artifact-annotated datasets. In this paper, we propose ArtiAgent, which efficiently creates pairs of real and artifact-injected images. It comprises three agents: a perception agent that recognizes and grounds entities and subentities from real images, a synthesis agent that introduces artifacts via artifact injection tools through novel patch-wise embedding manipulation within a diffusion transformer, and a curation agent that filters the synthesized artifacts and generates both local and global explanations for each instance. Using ArtiAgent, we synthesize 100K images with rich artifact annotations and demonstrate both efficacy and versatility across diverse applications. Code is available at link.
Paper Structure (47 sections, 12 equations, 20 figures, 7 tables, 1 algorithm)

This paper contains 47 sections, 12 equations, 20 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Understanding Visual Artifacts
  4. Problem Scope
  5. Artifact Analysis in Modern Generative Models
  6. Agentic Pipeline for Artifact Synthesis
  7. Perception Agent (Figure \ref{['fig:method']}a)
  8. Entity-Subentity Vocabulary Generation
  9. Entity-Subentity Grounding
  10. Synthesis Agent (Figure \ref{['fig:method']}b)
  11. Target-Reference Patch Mapping Toolbox
  12. Inversion-Injection Method for Artifact Synthesis
  13. Curation Agent (Figure \ref{['fig:method']}c)
  14. Data Filtering
  15. Explanation Generation
  16. ...and 32 more sections

Figures (20)

  • Figure 1: Overview of our challenges and approach. The red boxes indicate the regions with visual artifacts. (a) Examples of structural visual artifacts in state-of-the-art diffusion models and the inability of VLMs to recognize or explain them. (b) Overview of ArtiAgent, a novel agentic framework that synthesizes artifacts for arbitrary visual contexts without human intervention. (c) Example of VLM-based artifact comprehension via detection, explanation, and localization. (d) Application to reward-guided text-to-image generation. (e) Application to image correction, where artifact-aware VLM-guided inpainting removes the flawed regions.
  • Figure 1: Artifact frequency of modern diffusion models.
  • Figure 2: Artifact type distribution of diffusion models.
  • Figure 3: ArtiAgent consists of three coordinated agents: (1) the perception agent detects entities and subentities using Grounded-SAM; (2) the synthesis agent injects artifacts through patch mapping tool and the inversion-injection paradigm; and (3) the curation agent filters low-quality results and generates localized and global textual explanations.
  • Figure 4: Visualization of each target-reference patch mapping and its resulting artifact-injected image.
  • ...and 15 more figures

Theorems & Definitions (1)

  • Definition 3.1