Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PixLens: A Novel Framework for Disentangled Evaluation in Diffusion-Based Image Editing with Object Detection + SAM

Stefan Stefanache, Lluís Pastor Pérez, Julen Costa Watanabe, Ernesto Sanchez Tejedor, Thomas Hofmann, Enis Simsar

TL;DR

PixLens presents an automatic, task-aware benchmark for diffusion-based text-guided image editing that overcomes limitations of CLIP- and FID-centric evaluations by integrating SAM-based detection with segmentation masks to assess edit quality, subject preservation, and background fidelity. It introduces a nine-edit-type pipeline with continuous scoring, multiplicity handling, and a disentanglement analysis to understand how latent representations map to edits. The framework demonstrates stronger alignment with human judgments than prior benchmarks (e.g., EditVAL) and reveals that spatial edits remain challenging across models, while disentanglement metrics correlate with editing performance for several state-of-the-art methods. By preprocessing external datasets (MagicBrush) and providing detailed supplementary material, PixLens offers a scalable, component-wise approach that enables developers to diagnose and improve editing models in a realistic, post-edit evaluation setting.

Abstract

Evaluating diffusion-based image-editing models is a crucial task in the field of Generative AI. Specifically, it is imperative to assess their capacity to execute diverse editing tasks while preserving the image content and realism. While recent developments in generative models have opened up previously unheard-of possibilities for image editing, conducting a thorough evaluation of these models remains a challenging and open task. The absence of a standardized evaluation benchmark, primarily due to the inherent need for a post-edit reference image for evaluation, further complicates this issue. Currently, evaluations often rely on established models such as CLIP or require human intervention for a comprehensive understanding of the performance of these image editing models. Our benchmark, PixLens, provides a comprehensive evaluation of both edit quality and latent representation disentanglement, contributing to the advancement and refinement of existing methodologies in the field.

PixLens: A Novel Framework for Disentangled Evaluation in Diffusion-Based Image Editing with Object Detection + SAM

TL;DR

PixLens presents an automatic, task-aware benchmark for diffusion-based text-guided image editing that overcomes limitations of CLIP- and FID-centric evaluations by integrating SAM-based detection with segmentation masks to assess edit quality, subject preservation, and background fidelity. It introduces a nine-edit-type pipeline with continuous scoring, multiplicity handling, and a disentanglement analysis to understand how latent representations map to edits. The framework demonstrates stronger alignment with human judgments than prior benchmarks (e.g., EditVAL) and reveals that spatial edits remain challenging across models, while disentanglement metrics correlate with editing performance for several state-of-the-art methods. By preprocessing external datasets (MagicBrush) and providing detailed supplementary material, PixLens offers a scalable, component-wise approach that enables developers to diagnose and improve editing models in a realistic, post-edit evaluation setting.

Abstract

Evaluating diffusion-based image-editing models is a crucial task in the field of Generative AI. Specifically, it is imperative to assess their capacity to execute diverse editing tasks while preserving the image content and realism. While recent developments in generative models have opened up previously unheard-of possibilities for image editing, conducting a thorough evaluation of these models remains a challenging and open task. The absence of a standardized evaluation benchmark, primarily due to the inherent need for a post-edit reference image for evaluation, further complicates this issue. Currently, evaluations often rely on established models such as CLIP or require human intervention for a comprehensive understanding of the performance of these image editing models. Our benchmark, PixLens, provides a comprehensive evaluation of both edit quality and latent representation disentanglement, contributing to the advancement and refinement of existing methodologies in the field.
Paper Structure (37 sections, 1 equation, 22 figures, 6 tables, 8 algorithms)

This paper contains 37 sections, 1 equation, 22 figures, 6 tables, 8 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. PixLens: Benchmark for Text-Guided Image Editing
  4. Edit Quality Automatic Evaluation
  5. Evaluated Edit Types and Pipeline Structure.
  6. Detection and Segmentation Integration.
  7. Subject Preservation.
  8. Background Preservation.
  9. Disentanglement Evaluation
  10. Setup and Disentanglement Pipeline.
  11. Computing Scores.
  12. Automated Evaluation of State-of-the-Art Models
  13. Performance Overview and Analysis
  14. Disentanglement Analysis and Correlation
  15. Quantitative Analysis of the Benchmark
  16. ...and 22 more sections

Figures (22)

  • Figure 1: PixLens Edit Evaluation Pipeline: SIZE operation evaluation example.
  • Figure 2: Original image (left) and edited images resulting of "changing the color of the boat to black (center) / yellow (right)", using InstructPix2Pix.
  • Figure 3: Good subject preservation example for InstructPix2Pix edit with the prompt Change the color of the dog to red. The subject of this edit is the dog.
  • Figure 4: Bad subject preservation example for LCM edit with the prompt Change the color of the backpack to orange. The subject of this edit is the backpack.
  • Figure 5: Intra-attribute disentanglement example
  • ...and 17 more figures