AGHI-QA: A Subjective-Aligned Dataset and Metric for AI-Generated Human Images

TL;DR

This work fills a critical gap in evaluating AI-generated human images by introducing AGHI-QA, a large-scale, multidimensional benchmark tailored to human-centric distortions and text-image alignment. It proposes AGHI-Assessor, an LMM-based quality assessor that fuses visual encodings, human-centered cropping, AGI-aware quality signals, and text prompts to regress perceptual quality, TI correspondence, and to identify visible and distorted body parts. Extensive experiments on AGHI-QA and AGIQA-3K demonstrate state-of-the-art performance and strong generalization, with ablations validating the contribution of each component and training strategy (instruct tuning, LoRA adaptation, and loss design). The dataset and the method offer a practical framework for developing and evaluating finer-grained quality metrics for AGHIs, enabling better benchmarking of T2I models and more reliable detection of structural artifacts in human images.

Abstract

The rapid development of text-to-image (T2I) generation approaches has attracted extensive interest in evaluating the quality of generated images, leading to the development of various quality assessment methods for general-purpose T2I outputs. However, existing image quality assessment (IQA) methods are limited to providing global quality scores, failing to deliver fine-grained perceptual evaluations for structurally complex subjects like humans, which is a critical challenge considering the frequent anatomical and textural distortions in AI-generated human images (AGHIs). To address this gap, we introduce AGHI-QA, the first large-scale benchmark specifically designed for quality assessment of AGHIs. The dataset comprises 4,000 images generated from 400 carefully crafted text prompts using 10 state of-the-art T2I models. We conduct a systematic subjective study to collect multidimensional annotations, including perceptual quality scores, text-image correspondence scores, visible and distorted body part labels. Based on AGHI-QA, we evaluate the strengths and weaknesses of current T2I methods in generating human images from multiple dimensions. Furthermore, we propose AGHI-Assessor, a novel quality metric that integrates the large multimodal model (LMM) with domain-specific human features for precise quality prediction and identification of visible and distorted body parts in AGHIs. Extensive experimental results demonstrate that AGHI-Assessor showcases state-of-the-art performance, significantly outperforming existing IQA methods in multidimensional quality assessment and surpassing leading LMMs in detecting structural distortions in AGHIs.

Figures (8)

• Figure 1: Illustration of various distortion problems in AI-generated human images from current text-to-image approaches.
• Figure 2: The overview of our quality assessment dataset AGHI-QA. TI-correspondence denotes the score of text-image correspondence.
• Figure 3: Visualization of generated human-centric images from current popular text-to-image (T2I) methods in AGHI-QA dataset. We can observe the distinct characteristics of different T2I models. For instance, the latest models trends to generate more natural and high quality images than previous models. Playground v2.5 li2024playground is prone to synthesize aesthetic human images in terms of color and lighting, while Cosmicman li2024cosmicman and SD-XL dhariwal2021diffusion are prone to generate realistic human images.
• Figure 4: Visualization of MOS distribution and model-wise comparison results for AGHI-QA dataset.
• Figure 5: Dataset Distribution Analyze. The top left corner shows the distributions of text prompts under different categories. The rest area shows the detailed distributions of MOS scores across different combinations of attributes, different action classes and different scenarios.