Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset

Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, Eneko Agirre, Frank Keller

TL;DR

This work addresses the gap where text-to-image models struggle with explicit spatial relations due to training data lacking such cues. It introduces SR4G, a large automatically derived dataset containing 14 explicit spatial relations (9.9M training pairs and 60k+ evaluation captions) built from COCO via heuristic rules and templates. Fine-tuning Stable Diffusion models on SR4G yields substantial gains in spatial understanding (VISOR_Cond) and generalizes to unseen objects, outperforming state-of-the-art pipeline approaches with fewer parameters. The dataset and code are released to enable data-centric improvements in spatial reasoning for image generation and beyond.

Abstract

Existing work has observed that current text-to-image systems do not accurately reflect explicit spatial relations between objects such as 'left of' or 'below'. We hypothesize that this is because explicit spatial relations rarely appear in the image captions used to train these models. We propose an automatic method that, given existing images, generates synthetic captions that contain 14 explicit spatial relations. We introduce the Spatial Relation for Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for training, and more than 60 thousand captions for evaluation. In order to test generalization we also provide an 'unseen' split, where the set of objects in the train and test captions are disjoint. SR4G is the first dataset that can be used to spatially fine-tune text-to-image systems. We show that fine-tuning two different Stable Diffusion models (denoted as SD$_{SR4G}$) yields up to 9 points improvements in the VISOR metric. The improvement holds in the 'unseen' split, showing that SD$_{SR4G}$ is able to generalize to unseen objects. SD$_{SR4G}$ improves the state-of-the-art with fewer parameters, and avoids complex architectures. Our analysis shows that improvement is consistent for all relations. The dataset and the code will be publicly available.

Improving Explicit Spatial Relationships in Text-to-Image Generation through an Automatically Derived Dataset

TL;DR

This work addresses the gap where text-to-image models struggle with explicit spatial relations due to training data lacking such cues. It introduces SR4G, a large automatically derived dataset containing 14 explicit spatial relations (9.9M training pairs and 60k+ evaluation captions) built from COCO via heuristic rules and templates. Fine-tuning Stable Diffusion models on SR4G yields substantial gains in spatial understanding (VISOR_Cond) and generalizes to unseen objects, outperforming state-of-the-art pipeline approaches with fewer parameters. The dataset and code are released to enable data-centric improvements in spatial reasoning for image generation and beyond.

Abstract

Existing work has observed that current text-to-image systems do not accurately reflect explicit spatial relations between objects such as 'left of' or 'below'. We hypothesize that this is because explicit spatial relations rarely appear in the image captions used to train these models. We propose an automatic method that, given existing images, generates synthetic captions that contain 14 explicit spatial relations. We introduce the Spatial Relation for Generation (SR4G) dataset, which contains 9.9 millions image-caption pairs for training, and more than 60 thousand captions for evaluation. In order to test generalization we also provide an 'unseen' split, where the set of objects in the train and test captions are disjoint. SR4G is the first dataset that can be used to spatially fine-tune text-to-image systems. We show that fine-tuning two different Stable Diffusion models (denoted as SD) yields up to 9 points improvements in the VISOR metric. The improvement holds in the 'unseen' split, showing that SD is able to generalize to unseen objects. SD improves the state-of-the-art with fewer parameters, and avoids complex architectures. Our analysis shows that improvement is consistent for all relations. The dataset and the code will be publicly available.
Paper Structure (28 sections, 3 equations, 6 figures, 9 tables)

This paper contains 28 sections, 3 equations, 6 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SR4G: A new synthetic dataset for explicit spatial relation generation
  4. Captions for evaluation
  5. Image-caption pairs for training
  6. Dataset splits
  7. Evaluation metrics
  8. Object Accuracy:
  9. VISOR:
  10. VISOR$_{\mathrm{Cond}}$:
  11. Experiments and Results
  12. Experimental set-up
  13. Main results
  14. Comparison with the state of the art
  15. Analysis
  16. ...and 13 more sections

Figures (6)

  • Figure 1: Fine-tuning Stable Diffusion on our SR4G dataset improves results significantly (two versions of SD shown), surpassing the state of the art in spatial-aware systems (see Section \ref{['sec:exps']}).
  • Figure 2: The horizontal axis depicts the difference of VISOR$_\mathrm{Cond}$ values between relation pairs with opposing meanings defined on each side of the vertical axis. Results for SD and SD$_{SR4G}$ v2.1 on the unseen split.
  • Figure 3: Results using main splits.
  • Figure 4: Results using unseen splits.
  • Figure 6: Image generation examples by SD v2.1 and SD$_{SR4G}$ v2.1 fine-tuned on the main split. Following our relation-specific heuristics, if the relation in the caption is correctly depicted, we indicate this with a green tick. Otherwise, there is a red cross in the top-right corner of the image.
  • ...and 1 more figures