Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Scalable Human-aligned Benchmark for Text-guided Image Editing

Suho Ryu, Kihyun Kim, Eugene Baek, Dongsoo Shin, Joonseok Lee

TL;DR

This work addresses the lack of standardized evaluation for text-guided image editing by introducing HATIE, a large-scale, human-aligned benchmark with an automated evaluation pipeline. HATIE combines a GQA-derived dataset of 18,226 images with object-centric and non-object-centric edit queries and a five-criterion scoring framework (Image Quality, Object/Background Fidelity, Object/Background Consistency) whose weights are learned to align with human judgments. The authors validate human-alignment through a dedicated user study and demonstrate strong correlations between HATIE scores and human preferences across both description-based and instruction-based models, while revealing trade-offs between fidelity and consistency. The benchmark provides a reproducible protocol to fairly compare editing models and offers practical guidance for improving text-guided image editing systems.

Abstract

A variety of text-guided image editing models have been proposed recently. However, there is no widely-accepted standard evaluation method mainly due to the subjective nature of the task, letting researchers rely on manual user study. To address this, we introduce a novel Human-Aligned benchmark for Text-guided Image Editing (HATIE). Providing a large-scale benchmark set covering a wide range of editing tasks, it allows reliable evaluation, not limited to specific easy-to-evaluate cases. Also, HATIE provides a fully-automated and omnidirectional evaluation pipeline. Particularly, we combine multiple scores measuring various aspects of editing so as to align with human perception. We empirically verify that the evaluation of HATIE is indeed human-aligned in various aspects, and provide benchmark results on several state-of-the-art models to provide deeper insights on their performance.

Towards Scalable Human-aligned Benchmark for Text-guided Image Editing

TL;DR

This work addresses the lack of standardized evaluation for text-guided image editing by introducing HATIE, a large-scale, human-aligned benchmark with an automated evaluation pipeline. HATIE combines a GQA-derived dataset of 18,226 images with object-centric and non-object-centric edit queries and a five-criterion scoring framework (Image Quality, Object/Background Fidelity, Object/Background Consistency) whose weights are learned to align with human judgments. The authors validate human-alignment through a dedicated user study and demonstrate strong correlations between HATIE scores and human preferences across both description-based and instruction-based models, while revealing trade-offs between fidelity and consistency. The benchmark provides a reproducible protocol to fairly compare editing models and offers practical guidance for improving text-guided image editing systems.

Abstract

A variety of text-guided image editing models have been proposed recently. However, there is no widely-accepted standard evaluation method mainly due to the subjective nature of the task, letting researchers rely on manual user study. To address this, we introduce a novel Human-Aligned benchmark for Text-guided Image Editing (HATIE). Providing a large-scale benchmark set covering a wide range of editing tasks, it allows reliable evaluation, not limited to specific easy-to-evaluate cases. Also, HATIE provides a fully-automated and omnidirectional evaluation pipeline. Particularly, we combine multiple scores measuring various aspects of editing so as to align with human perception. We empirically verify that the evaluation of HATIE is indeed human-aligned in various aspects, and provide benchmark results on several state-of-the-art models to provide deeper insights on their performance.
Paper Structure (38 sections, 11 equations, 23 figures, 9 tables)

This paper contains 38 sections, 11 equations, 23 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. The HATIE Framework
  4. Dataset
  5. Editing Query Formulation
  6. Evaluation Criteria and Metrics
  7. Human-aligned Benchmark
  8. Empirical Verification
  9. Benchmark with Varied Edit Intensity
  10. Alignment to Human Perception
  11. HATIE Benchmark Results and Discussion
  12. Summary and Limitations
  13. More Details on Dataset Filtering
  14. More Details on Editing Query
  15. Details on Generation Rules
  16. ...and 23 more sections

Figures (23)

  • Figure 1: An example highlighting the importance of consistency in image editing. (a) Original image (b) Edited images for a prompt "Make her smile". The left result is more consistent with the input than the right one, better preserving her identity.
  • Figure 2: Overview of our HATIE Benchmark. HATIE consists of an image and query dataset for editing, along with an automated evaluation pipeline for assessing editing performance. We curate a large-scale comprehensive dataset with images and corresponding editing queries, on which a model would perform text-guided image editing. Then, HATIE evaluates the edited images from 5 different aspects: Object Fidelity, Background Fidelity, Object Consistency, Background Consistency, and Image Quality. Finally, these scores are aggregated by a weight fitted to human feedback through our user study, producing the final Total Score.
  • Figure 3: Object Class Distribution in Our Dataset. HATIE evaluates fairly by providing evenly distributed dataset.
  • Figure 4: Query Set Distribution. (a) Distribution of edit types in our query set, (b) Distribution of the object classes designated as the target in object-centric queries.
  • Figure 5: Evaluation Workflow Specific to Each Editing Task. See \ref{['sec:eval_workflow']} for more details.
  • ...and 18 more figures