Evaluating Text-to-Image Generative Models: An Empirical Study on Human Image Synthesis

TL;DR

An innovative aesthetic score prediction model is introduced that assesses the visual appeal of generated images and unveils the first dataset marked with low-quality regions in generated human images to facilitate automatic defect detection.

Abstract

In this paper, we present an empirical study introducing a nuanced evaluation framework for text-to-image (T2I) generative models, applied to human image synthesis. Our framework categorizes evaluations into two distinct groups: first, focusing on image qualities such as aesthetics and realism, and second, examining text conditions through concept coverage and fairness. We introduce an innovative aesthetic score prediction model that assesses the visual appeal of generated images and unveils the first dataset marked with low-quality regions in generated human images to facilitate automatic defect detection. Our exploration into concept coverage probes the model's effectiveness in interpreting and rendering text-based concepts accurately, while our analysis of fairness reveals biases in model outputs, with an emphasis on gender, race, and age. While our study is grounded in human imagery, this dual-faceted approach is designed with the flexibility to be applicable to other forms of image generation, enhancing our understanding of generative models and paving the way to the next generation of more sophisticated, contextually aware, and ethically attuned generative models. Code and data, including the dataset annotated with defective areas, are available at \href{https://github.com/cure-lab/EvaluateAIGC}{https://github.com/cure-lab/EvaluateAIGC}.

Figures (8)

• Figure 1: An overview of our framework. Given a T2I generative model for evaluation, we first generate prompts from diverse human-related concepts. Then, we use the T2I generative model to generate images. Each generated image undergoes a dual evaluation process: 1) its aesthetic appeal is assessed using our novel aesthetic score prediction model, CAN, and 2) its realism is evaluated by models trained on our annotated defect dataset. For all images associated with a specific concept, 3) our VQA-based methodologies are employed to discern the absence of biases, and 4) our proposed concept coverage metrics are utilized to assess the fidelity of concept depiction.
• Figure 2: An overview of CAN. CAN predicts scores for the general aesthetic feeling and specific aesthetic attributes of an image. In the training process, the overall model is trained to predict aesthetic scores. Additionally, the generic aesthetic module is trained to predict the distortion applied to an image. $\oplus$ denotes concatenation.
• Figure 3: The rankings of eight human images by CAN and TANet compared with the ground truth. Red color indicates incorrect rankings.
• Figure 4: Bounding boxes and label assignments for a generated human image. The face box is highlighted in yellow, while the human body box is depicted in white. The table shows annotations of both component-wise and coarse-grained labels. The red bounding box denotes areas with defects.
• Figure 5: Facial defects from Midjourney, as identified by our face evaluation model.