IMDL-BenCo: A Comprehensive Benchmark and Codebase for Image Manipulation Detection & Localization

TL;DR

IMDL-BenCo is introduced, the first comprehensive IMDL benchmark and modular codebase that decomposes the IMDL framework into standardized, reusable components and revises the model construction pipeline, improving coding efficiency and customization flexibility.

Abstract

A comprehensive benchmark is yet to be established in the Image Manipulation Detection & Localization (IMDL) field. The absence of such a benchmark leads to insufficient and misleading model evaluations, severely undermining the development of this field. However, the scarcity of open-sourced baseline models and inconsistent training and evaluation protocols make conducting rigorous experiments and faithful comparisons among IMDL models challenging. To address these challenges, we introduce IMDL-BenCo, the first comprehensive IMDL benchmark and modular codebase. IMDL-BenCo: i) decomposes the IMDL framework into standardized, reusable components and revises the model construction pipeline, improving coding efficiency and customization flexibility; ii) fully implements or incorporates training code for state-of-the-art models to establish a comprehensive IMDL benchmark; and iii) conducts deep analysis based on the established benchmark and codebase, offering new insights into IMDL model architecture, dataset characteristics, and evaluation standards. Specifically, IMDL-BenCo includes common processing algorithms, 8 state-of-the-art IMDL models (1 of which are reproduced from scratch), 2 sets of standard training and evaluation protocols, 15 GPU-accelerated evaluation metrics, and 3 kinds of robustness evaluation. This benchmark and codebase represent a significant leap forward in calibrating the current progress in the IMDL field and inspiring future breakthroughs. Code is available at: https://github.com/scu-zjz/IMDLBenCo.

Figures (7)

• Figure 1: Overview of the paradigm for IMDL-BenCo.
• Figure 2: Results for robustness evaluation under Protocol-MVSS.
• Figure 3: An instance of label leakage in NIST16. There is almost no visible difference between these images and their masks. When split into training and testing datasets, the test images are easily located by the model, indicating an overestimation of model performance.
• Figure 4: Controversial permuted metrics. When the model's predictions are completely wrong, the F1 score should theoretically be 0.00.
• Figure 5: Grad-CAM visualization of models in our benchmark