Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

X-Edit: Detecting and Localizing Edits in Images Altered by Text-Guided Diffusion Models

Valentina Bazyleva, Nicolo Bonettini, Gaurav Bharaj

TL;DR

This work tackles the challenge of localizing edits made by text-guided diffusion models. It proposes X-Edit, an inversion-based segmentation framework that leverages discrepancies from a pretrained diffusion model to predict edited regions, enhanced by a CBAM-U-Net and a RobustViT-inspired finetuning with a Sobel-based frequency-aware relevance loss. A new large paired dataset of original and edited images (167,026 pairs) is introduced to train and evaluate localization, with ground-truth masks derived from pixel-wise differences. Experimental results show that X-Edit outperforms baselines in PSNR and SSIM, achieving precise, edge-aligned localization of edits and offering a robust forensic tool for detecting and pinpointing diffusion-based manipulations.

Abstract

Text-guided diffusion models have significantly advanced image editing, enabling highly realistic and local modifications based on textual prompts. While these developments expand creative possibilities, their malicious use poses substantial challenges for detection of such subtle deepfake edits. To this end, we introduce Explain Edit (X-Edit), a novel method for localizing diffusion-based edits in images. To localize the edits for an image, we invert the image using a pretrained diffusion model, then use these inverted features as input to a segmentation network that explicitly predicts the edited masked regions via channel and spatial attention. Further, we finetune the model using a combined segmentation and relevance loss. The segmentation loss ensures accurate mask prediction by balancing pixel-wise errors and perceptual similarity, while the relevance loss guides the model to focus on low-frequency regions and mitigate high-frequency artifacts, enhancing the localization of subtle edits. To the best of our knowledge, we are the first to address and model the problem of localizing diffusion-based modified regions in images. We additionally contribute a new dataset of paired original and edited images addressing the current lack of resources for this task. Experimental results demonstrate that X-Edit accurately localizes edits in images altered by text-guided diffusion models, outperforming baselines in PSNR and SSIM metrics. This highlights X-Edit's potential as a robust forensic tool for detecting and pinpointing manipulations introduced by advanced image editing techniques.

X-Edit: Detecting and Localizing Edits in Images Altered by Text-Guided Diffusion Models

TL;DR

This work tackles the challenge of localizing edits made by text-guided diffusion models. It proposes X-Edit, an inversion-based segmentation framework that leverages discrepancies from a pretrained diffusion model to predict edited regions, enhanced by a CBAM-U-Net and a RobustViT-inspired finetuning with a Sobel-based frequency-aware relevance loss. A new large paired dataset of original and edited images (167,026 pairs) is introduced to train and evaluate localization, with ground-truth masks derived from pixel-wise differences. Experimental results show that X-Edit outperforms baselines in PSNR and SSIM, achieving precise, edge-aligned localization of edits and offering a robust forensic tool for detecting and pinpointing diffusion-based manipulations.

Abstract

Text-guided diffusion models have significantly advanced image editing, enabling highly realistic and local modifications based on textual prompts. While these developments expand creative possibilities, their malicious use poses substantial challenges for detection of such subtle deepfake edits. To this end, we introduce Explain Edit (X-Edit), a novel method for localizing diffusion-based edits in images. To localize the edits for an image, we invert the image using a pretrained diffusion model, then use these inverted features as input to a segmentation network that explicitly predicts the edited masked regions via channel and spatial attention. Further, we finetune the model using a combined segmentation and relevance loss. The segmentation loss ensures accurate mask prediction by balancing pixel-wise errors and perceptual similarity, while the relevance loss guides the model to focus on low-frequency regions and mitigate high-frequency artifacts, enhancing the localization of subtle edits. To the best of our knowledge, we are the first to address and model the problem of localizing diffusion-based modified regions in images. We additionally contribute a new dataset of paired original and edited images addressing the current lack of resources for this task. Experimental results demonstrate that X-Edit accurately localizes edits in images altered by text-guided diffusion models, outperforming baselines in PSNR and SSIM metrics. This highlights X-Edit's potential as a robust forensic tool for detecting and pinpointing manipulations introduced by advanced image editing techniques.
Paper Structure (24 sections, 9 equations, 13 figures, 2 tables)

This paper contains 24 sections, 9 equations, 13 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Image-to-Image Translation in the Context of Diffusion Models
  4. Detectors of Diffusion-based Generated Images
  5. Detection and XAI Methods for Images Edited with Image-to-Image Diffusion Models
  6. Method
  7. Dataset
  8. Feature Extraction
  9. Edit Localization with X-Edit
  10. Finetuning Procedure
  11. Experiments
  12. Input Data Format
  13. Model Architecture
  14. Comparison Models
  15. Inference and Evaluation Metrics
  16. ...and 9 more sections

Figures (13)

  • Figure 1: Examples of our X-Edit localization method. X-Edit is able to localize text-guided image edits disregarding areas not affected by the edit.
  • Figure 2: Grad-CAM visualizations for original and edited images. The results highlight that the coarse relevance maps produced by EfficientNet with high detection accuracy (99.93%) and precision (99.92%) do not accurately localize specific edit regions.
  • Figure 3: Overview of the X-Edit framework. The pipeline begins with paired data creation, where each image is modified with InstructPix2Pix based on one specified edit caption to produce an edited image. A ground-truth mask is generated by computing the absolute difference between original and corresponding edited images. In the X-Edit, input features, including concatenated image, decoded noise, reconstructed image, and residual differences, are processed by a U-Net architecture with integrated CBAM blocks for enhanced attention. The model is optimized using a combined loss to generate an output mask that reconstructs the ground-truth mask, highlighting edited regions.
  • Figure 4: Comparison of predicted masks for edited images. From left to right: original image, edited image, ground truth mask indicating the edited regions, predicted mask from X-Edit finetuned on $\bm{\phi}$, X-Edit on $\bm{\phi}_{\textrm{FI}}$, SAM and SegFormer. X-Edit finetuned on $\bm{\phi}$ (4th column) outperforms other models by more accurately capturing both the shape and placement of edits, demonstrating finer boundary alignment and better preservation of details in complex regions. This improvement highlights X-Edit's effectiveness in maintaining contextual coherence and producing higher-fidelity masks for intricate modifications.
  • Figure 5: Comparison of predicted mask for original images. X-Edit manages to predict almost blank masks for original images, while SAM tends to produce false positives.
  • ...and 8 more figures