Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Mapping the Mind of an Instruction-based Image Editing using SMILE

Zeinab Dehghani, Koorosh Aslansefat, Adil Khan, Adín Ramírez Rivera, Franky George, Muhammad Khalid

TL;DR

Instruction-based image editing models remain black boxes, limiting transparency in high-stakes domains. We introduce SMILE, a model-agnostic, localized interpretability framework that builds perturbed prompts, maps edits to a 1D distance via Wasserstein-based metrics, and uses a weighted linear surrogate to produce word-level heatmaps. SMILE couples DINOv2 embeddings with bootstrap p-values to assess significance and demonstrates robust accuracy, stability, and fidelity across multiple diffusion models, including Instruct-Pix2Pix, Img2Img-Turbo, and Diffusers-Inpaint. The results indicate that SMILE enhances trust and controllability of text-driven edits, with practical implications for healthcare, autonomous driving, and other precision editing contexts; the work lays a foundation for extending model-agnostic interpretability to broader generative tasks.

Abstract

Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as healthcare and autonomous driving while encouraging additional investigation into the significance of interpretability in enhancing dependable image editing models.

Mapping the Mind of an Instruction-based Image Editing using SMILE

TL;DR

Instruction-based image editing models remain black boxes, limiting transparency in high-stakes domains. We introduce SMILE, a model-agnostic, localized interpretability framework that builds perturbed prompts, maps edits to a 1D distance via Wasserstein-based metrics, and uses a weighted linear surrogate to produce word-level heatmaps. SMILE couples DINOv2 embeddings with bootstrap p-values to assess significance and demonstrates robust accuracy, stability, and fidelity across multiple diffusion models, including Instruct-Pix2Pix, Img2Img-Turbo, and Diffusers-Inpaint. The results indicate that SMILE enhances trust and controllability of text-driven edits, with practical implications for healthcare, autonomous driving, and other precision editing contexts; the work lays a foundation for extending model-agnostic interpretability to broader generative tasks.

Abstract

Despite recent advancements in Instruct-based Image Editing models for generating high-quality images, they are known as black boxes and a significant barrier to transparency and user trust. To solve this issue, we introduce SMILE (Statistical Model-agnostic Interpretability with Local Explanations), a novel model-agnostic for localized interpretability that provides a visual heatmap to clarify the textual elements' influence on image-generating models. We applied our method to various Instruction-based Image Editing models like Pix2Pix, Image2Image-turbo and Diffusers-Inpaint and showed how our model can improve interpretability and reliability. Also, we use stability, accuracy, fidelity, and consistency metrics to evaluate our method. These findings indicate the exciting potential of model-agnostic interpretability for reliability and trustworthiness in critical applications such as healthcare and autonomous driving while encouraging additional investigation into the significance of interpretability in enhancing dependable image editing models.

Paper Structure

This paper contains 31 sections, 14 equations, 17 figures, 11 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Literature review
  3. Instruction-Based Image Editing Diffusion models
  4. Explainable Artificial Intelligence
  5. Explainable diffusion based image editing
  6. Problem Definition
  7. Proposed Method
  8. Image Generating and perturbed prompts
  9. Creating the interpretable space
  10. Developing the Interpretable Surrogate Model
  11. Evaluation Metrics
  12. Accuracy
  13. Stability
  14. Consistency
  15. Fidelity
  16. ...and 16 more sections

Figures (17)

  • Figure 1: Overview of SMILE for explaining instruct image editing diffusion-based models
  • Figure 2: Overview of Variations and Enhancements in the LIME (Local Interpretable Model-Agnostic Explanations) Framework. This figure illustrates key modifications to LIME across different aspects: surrogate models, distance metrics, sampling techniques, optimization methods, and adaptations for specific data types.
  • Figure 3: 2D t-SNE visualization of image embeddings, showing distinct clusters for different prompt keywords, with clear separability between groups representing specific text features and those with perturbed, non-specific prompts.
  • Figure 4: SMILE flowchart for explaining image classification aslansefat2023explaining
  • Figure 5: SMILE flowchart for explaining instruct based image editing models
  • ...and 12 more figures