Webly-Supervised Image Manipulation Localization via Category-Aware Auto-Annotation

TL;DR

Data scarcity for image manipulation localization (IML) limits model generalization. The authors propose a web-scale supervision pipeline built around Category-Aware Auto-Annotation v2 (CAAAv2), a prior-denoising CIML framework, to auto-annotate web-forgeries and create large, diverse training data (MIMLv2). They introduce QES to filter annotations and Object Jitter to enrich hard cases, then fuse these signals in Web-IML, a model with Multi-Scale Perception and Self-Rectification. Empirical results show Web-IML achieves a 31% relative performance gain and surpasses the SparseViT state of the art by 21.6 average IoU points, on multiple forgery benchmarks and a downstream document IML task. Altogether, the work provides a scalable, data-driven path to continuous web-based data expansion for robust IML systems.

Abstract

Images manipulated by image editing tools can mislead viewers and pose significant risks to social security. However, accurately localizing manipulated image regions remains challenging due to the severe scarcity of high-quality annotated data, which is laborious to create. To address this, we propose a novel approach that mitigates data scarcity by leveraging readily available web data. We utilize a large collection of manually forged images from the web, as well as automatically generated annotations derived from a simpler auxiliary task, constrained image manipulation localization.Specifically, we introduce CAAAv2, a novel auto-annotation framework that operates on a category-aware, prior-feature-denoising paradigm that notably reduces task complexity. To further ensure annotation reliability, we propose QES, a novel metric that filters out low-quality annotations. Combining CAAAv2 and QES, we construct MIMLv2, a large-scale, diverse, and high-quality dataset containing 246,212 manually forged images with pixel-level mask annotations. This is over 120 times larger than existing handcrafted datasets like IMD20. Additionally, we introduce Object Jitter, a technique that further enhances model training by generating high-quality manipulation artifacts. Building on these advances, we develop Web-IML, a new model designed to effectively leverage web-scale supervision for the task of image manipulation localization. Extensive experiments demonstrate that our approach substantially alleviates the data scarcity problem and significantly improves the performance of various models on multiple real-world forgery benchmarks. With the proposed web supervision, our Web-IML achieves a striking performance gain of 31% and surpasses the previous state-of-the-art SparseViT by 21.6 average IoU points. The dataset and code will be released at https://github.com/qcf-568/MIML.

Figures (11)

• Figure 1: Left: Definition and characteristics of the Shared Donor Group (SDG) and Shared Probe Group (SPG). Right: In contrast to previous works, our CAAAv2 adopts category-aware and prior-feature-denoising paradigm that notably reduces task difficulty and overfitting. We also propose using the CAAAv2 models to auto-annotate web-scale manually forged images. This approach significantly addresses the data scarcity issue for IML and notably improves the generalization of all IML models.
• Figure 2: The overall pipeline of our proposed CAAAv2 paradigm. The input original-forged image pairs are first categorized as either SDG or SPG with a classifier. Then, we use Corr-DINO and Difference-Aware Semantic Segmentation to process SDG and SPG pairs respectively. Both models first extract a prior feature that roughly highlights forgeries and then denoise it.
• Figure 3: The overall pipeline of our proposed Corr-DINO. First, we employ a frozen DINO backbone to extract image features. The features obtained from the final DINO layer are then used for correlation calculation. Next, a learnable aggregation module reduces the channel dimension of the correlation features, which are further refined using a Feature Super Resolution module. Finally, the refined features are denoised with a Multi-Aspect Denoiser, resulting in the final mask prediction. Green lines indicate operations used only during training with synthetic data, due to the lack of mask labels for the real images.
• Figure 4: Manipulated images often degrade during transmission. As a result, the absolute difference between a tampered image and its original does not accurately indicate the forged region. Our method effectively denoises the difference map by leveraging semantic information.
• Figure 5: The construction pipeline of our MIMLv2 dataset.