Cost Savings from Automatic Quality Assessment of Generated Images

TL;DR

The paper tackles the high cost of manual image quality assessment in automatic image generation pipelines by introducing a closed-form cost-estimation formula parameterized by AutoQA yield and precision. It frames a three-stage workflow (GenAI, AutoQA, ManualQA) and derives expressions for total cost and savings, validated on a background inpainting use case. Through a large, defect-annotated dataset, it compares AutoGluon-based AutoQA with a VLM-based alternative, showing substantial cost savings (up to 51.61%) when AutoQA effectively filters outputs before manual review, while highlighting limitations of zero-shot VLM approaches. The work suggests that improving generators and AutoQA components can further reduce or even eliminate manual review, with practical implications for scalable e-commerce imagery production.

Abstract

Deep generative models have shown impressive progress in recent years, making it possible to produce high quality images with a simple text prompt or a reference image. However, state of the art technology does not yet meet the quality standards offered by traditional photographic methods. For this reason, production pipelines that use generated images often include a manual stage of image quality assessment (IQA). This process is slow and expensive, especially because of the low yield of automatically generated images that pass the quality bar. The IQA workload can be reduced by introducing an automatic pre-filtering stage, that will increase the overall quality of the images sent to review and, therefore, reduce the average cost required to obtain a high quality image. We present a formula that estimates the cost savings depending on the precision and pass yield of a generic IQA engine. This formula is applied in a use case of background inpainting, showcasing a significant cost saving of 51.61% obtained with a simple AutoML solution.

Figures (11)

• Figure 1: Vertical pipeline for image generation and quality assessment with sample images. $N_{Gen}$, $N_{AQA}$ and $N_{MQA}$ represent the number of images output after each block. TP/FP/TN/FN refer to true/false positive/negatives.
• Figure 2: Types of visual defects in background inpainting. Top: Reference object. Bottom: Generated image.
• Figure 3: Distribution of defect severity across the six types of coarse defects. Each of the 8,304 generated images was manually labeled by a human annotator according to a scale of three levels: No defect, Some defect, or Significant defect. This figure does not consider agreement among annotators
• Figure 4: Histogram by object category of the train and test partition used to in the All Defects experiment.
• Figure 5: Volume of images needed after each block to obtain 100 high quality images.