Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Offline Evaluation of Set-Based Text-to-Image Generation

Negar Arabzadeh, Fernando Diaz, Junfeng He

TL;DR

This work develops TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings.

Abstract

Text-to-Image (TTI) systems often support people during ideation, the early stages of a creative process when exposure to a broad set of relevant images can help explore the design space. Since ideation is an important subclass of TTI tasks, understanding how to quantitatively evaluate TTI systems according to how well they support ideation is crucial to promoting research and development for these users. However, existing evaluation metrics for TTI remain focused on distributional similarity metrics like Fréchet Inception Distance (FID). We take an alternative approach and, based on established methods from ranking evaluation, develop TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images. Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images. We analyze our proposed family of TTI metrics using human studies on image grids generated by three different TTI systems based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings. Our results demonstrate that grounding metrics in how people use systems is an important and understudied area of benchmark design.

Offline Evaluation of Set-Based Text-to-Image Generation

TL;DR

This work develops TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings.

Abstract

Text-to-Image (TTI) systems often support people during ideation, the early stages of a creative process when exposure to a broad set of relevant images can help explore the design space. Since ideation is an important subclass of TTI tasks, understanding how to quantitatively evaluate TTI systems according to how well they support ideation is crucial to promoting research and development for these users. However, existing evaluation metrics for TTI remain focused on distributional similarity metrics like Fréchet Inception Distance (FID). We take an alternative approach and, based on established methods from ranking evaluation, develop TTI evaluation metrics with explicit models of how users browse and interact with sets of spatially arranged generated images. Our proposed offline evaluation metrics for TTI not only capture how relevant generated images are with respect to the user's ideation need but also take into consideration the diversity and arrangement of the set of generated images. We analyze our proposed family of TTI metrics using human studies on image grids generated by three different TTI systems based on subsets of the widely used benchmarks such as MS-COCO captions and Localized Narratives as well as prompts used in naturalistic settings. Our results demonstrate that grounding metrics in how people use systems is an important and understudied area of benchmark design.

Paper Structure

This paper contains 20 sections, 4 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Method
  4. Problem Definition
  5. Designing a Metric for TTI
  6. Experimental Setup
  7. TTI Systems
  8. Datasets
  9. Metric Variations
  10. Baseline TTI Evaluation Metrics
  11. Modeling Salience
  12. Data Annotation
  13. Annotation Process
  14. Agreement Rate
  15. Results and Validation
  16. ...and 5 more sections

Figures (3)

  • Figure 1: Overview of our proposed approach
  • Figure 2: Four image grids and associated salience maps for the prompt 'a cute puppy is painting'. Images grids in the left column have lower diversity than those in the right column as reflected by the different positions and breeds on puppies. In image grids in the top row, images tend decrease in salience from left to right; in the bottom row, there is less of a relationship between position and salience.
  • Figure 3: Annotation results of TTI systems A (orange), B (blue) and B' (green) on the three datasets. Given the TTI systems X/Y are on the left/right side of the arrows beneath a sub-figure, the bars in the sub-figure present if "X is much better than Y", "X is somewhat better than Y", "X and Y are the same", "Y is better than X" and "Y is much better than X" consecutively.