IML-ViT: Benchmarking Image Manipulation Localization by Vision Transformer

TL;DR

This work introduces IML-ViT, the first Vision Transformer-based approach for Image Manipulation Localization that preserves high-resolution input, applies multi-scale supervision via SFPN, and employs edge-focused supervision. Leveraging MAE pretraining, windowed attention, and a lightweight MLP head, it targets non-semantic artifact localization along manipulation borders. Across three mainstream evaluation protocols, IML-ViT delivers state-of-the-art F1 and AUC scores and shows strong robustness to common image corruptions. The study also provides a clear benchmark framework and ablation evidence that high resolution, multi-scale supervision, and edge guidance are key to effective IML with transformers.

Abstract

Advanced image tampering techniques are increasingly challenging the trustworthiness of multimedia, leading to the development of Image Manipulation Localization (IML). But what makes a good IML model? The answer lies in the way to capture artifacts. Exploiting artifacts requires the model to extract non-semantic discrepancies between manipulated and authentic regions, necessitating explicit comparisons between the two areas. With the self-attention mechanism, naturally, the Transformer should be a better candidate to capture artifacts. However, due to limited datasets, there is currently no pure ViT-based approach for IML to serve as a benchmark, and CNNs dominate the entire task. Nevertheless, CNNs suffer from weak long-range and non-semantic modeling. To bridge this gap, based on the fact that artifacts are sensitive to image resolution, amplified under multi-scale features, and massive at the manipulation border, we formulate the answer to the former question as building a ViT with high-resolution capacity, multi-scale feature extraction capability, and manipulation edge supervision that could converge with a small amount of data. We term this simple but effective ViT paradigm IML-ViT, which has significant potential to become a new benchmark for IML. Extensive experiments on three different mainstream protocols verified our model outperforms the state-of-the-art manipulation localization methods. Code and models are available at https://github.com/SunnyHaze/IML-ViT.

Figures (16)

• Figure 1: An example of three types of manipulations and their corresponding artifacts. Artifacts contain visible traces, including distortions, sudden changes, or anomalies caused by tampering operations. Artifacts are frequently found at the junction between two regions and appear in very detailed positions. For a better view, zooming in is recommended.
• Figure 2: Overview of the general structure of IML-ViT.
• Figure 3: Diagrams of the predict-head. The rectangles on the left represent the output of SFPN. There is a normalization layer before entering the MLP block, which is fully discussed below.
• Figure 4: Examples of generating the edge mask $M^\star$. White region represents for manipulated area, $k$ is set to 7 while the image size is 1024×682. The absolute value operation ensures that whether the tampered region dominates or the non-tampered region dominates, the mask only emphasizes the junction of the two.
• Figure 5: Lambda selection, trained/test on CASIAv2/v1.