Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Action and Reasoning-Centric Image Editing from Videos and Simulations

Benno Krojer, Dheeraj Vattikonda, Luis Lara, Varun Jampani, Eva Portelance, Christopher Pal, Siva Reddy

TL;DR

This work meticulously curates the AURORA Dataset (Action-Reasoning-Object-Attribute), a collection of high-quality training data, human-annotated and curated from videos and simulation engines, and proposes a new automatic metric that focuses on discriminative understanding.

Abstract

An image editing model should be able to perform diverse edits, ranging from object replacement, changing attributes or style, to performing actions or movement, which require many forms of reasoning. Current general instruction-guided editing models have significant shortcomings with action and reasoning-centric edits. Object, attribute or stylistic changes can be learned from visually static datasets. On the other hand, high-quality data for action and reasoning-centric edits is scarce and has to come from entirely different sources that cover e.g. physical dynamics, temporality and spatial reasoning. To this end, we meticulously curate the AURORA Dataset (Action-Reasoning-Object-Attribute), a collection of high-quality training data, human-annotated and curated from videos and simulation engines. We focus on a key aspect of quality training data: triplets (source image, prompt, target image) contain a single meaningful visual change described by the prompt, i.e., truly minimal changes between source and target images. To demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks. Our model significantly outperforms previous editing models as judged by human raters. For automatic evaluations, we find important flaws in previous metrics and caution their use for semantically hard editing tasks. Instead, we propose a new automatic metric that focuses on discriminative understanding. We hope that our efforts : (1) curating a quality training dataset and an evaluation benchmark, (2) developing critical evaluations, and (3) releasing a state-of-the-art model, will fuel further progress on general image editing.

Learning Action and Reasoning-Centric Image Editing from Videos and Simulations

TL;DR

This work meticulously curates the AURORA Dataset (Action-Reasoning-Object-Attribute), a collection of high-quality training data, human-annotated and curated from videos and simulation engines, and proposes a new automatic metric that focuses on discriminative understanding.

Abstract

An image editing model should be able to perform diverse edits, ranging from object replacement, changing attributes or style, to performing actions or movement, which require many forms of reasoning. Current general instruction-guided editing models have significant shortcomings with action and reasoning-centric edits. Object, attribute or stylistic changes can be learned from visually static datasets. On the other hand, high-quality data for action and reasoning-centric edits is scarce and has to come from entirely different sources that cover e.g. physical dynamics, temporality and spatial reasoning. To this end, we meticulously curate the AURORA Dataset (Action-Reasoning-Object-Attribute), a collection of high-quality training data, human-annotated and curated from videos and simulation engines. We focus on a key aspect of quality training data: triplets (source image, prompt, target image) contain a single meaningful visual change described by the prompt, i.e., truly minimal changes between source and target images. To demonstrate the value of our dataset, we evaluate an AURORA-finetuned model on a new expert-curated benchmark (AURORA-Bench) covering 8 diverse editing tasks. Our model significantly outperforms previous editing models as judged by human raters. For automatic evaluations, we find important flaws in previous metrics and caution their use for semantically hard editing tasks. Instead, we propose a new automatic metric that focuses on discriminative understanding. We hope that our efforts : (1) curating a quality training dataset and an evaluation benchmark, (2) developing critical evaluations, and (3) releasing a state-of-the-art model, will fuel further progress on general image editing.
Paper Structure (103 sections, 1 equation, 39 figures, 5 tables)

This paper contains 103 sections, 1 equation, 39 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Typology of image edits
  3. AURORA: A diverse and high quality image editing dataset
  4. Truly Minimal Visual Change
  5. Creating quality data for action-centric and reasoning-centric edits
  6. AURORA-Bench: A holistic editing benchmark
  7. Evaluation
  8. Evaluation of final generations
  9. DiscEdit: discriminative evaluation of image editing
  10. Results
  11. Human Judgement
  12. DiscEdit
  13. Ablations and qualitative analysis
  14. Conclusion and Future Work
  15. Dataset Release
  16. ...and 88 more sections

Figures (39)

  • Figure 1: Previous failures on editing skills such as action, movement and reasoning (measured in AURORA-Bench) compared to improvements with AURORA on these more challenging actions.
  • Figure 2: Our AURORA dataset covers action, reasoning and object-centric edits via 4 sub-datasets.
  • Figure 3: Common failure mode of previous models trained on noisy image pairs (e.g. InstructPix2Pix and GenHowTo): Their outputs are rarely faithful to the source image due to its noisy training pairs.
  • Figure 4: Prompt:Put the white porcelain ramekin on the right hand of the brown shoe. We show attention maps for three levels of U-Net layers: Down, Middle, Upper.
  • Figure 5: Object-centric edit example: Can we have a wooden table?
  • ...and 34 more figures