Mesoscopic Insights: Orchestrating Multi-scale & Hybrid Architecture for Image Manipulation Localization

TL;DR

The paper reframes image manipulation localization by introducing a mesoscopic perspective that fuses microscopic artifacts with macroscopic semantics. It proposes Mesorch, a parallel CNN-Transformer architecture that orchestrates multi-scale representations, enhanced by a frequency-based (DCT) feature integration, adaptive scale weighting, and scale pruning to improve accuracy and efficiency. Through extensive experiments on four benchmarks, Mesorch achieves state-of-the-art F1 scores, robustness under perturbations, and reduced computational cost relative to prior methods, with two practical baselines derived from the architecture. This work advances forensic localization by enabling robust, scalable, and interpretable mesoscopic artifact detection suitable for real-world deployments.

Abstract

The mesoscopic level serves as a bridge between the macroscopic and microscopic worlds, addressing gaps overlooked by both. Image manipulation localization (IML), a crucial technique to pursue truth from fake images, has long relied on low-level (microscopic-level) traces. However, in practice, most tampering aims to deceive the audience by altering image semantics. As a result, manipulation commonly occurs at the object level (macroscopic level), which is equally important as microscopic traces. Therefore, integrating these two levels into the mesoscopic level presents a new perspective for IML research. Inspired by this, our paper explores how to simultaneously construct mesoscopic representations of micro and macro information for IML and introduces the Mesorch architecture to orchestrate both. Specifically, this architecture i) combines Transformers and CNNs in parallel, with Transformers extracting macro information and CNNs capturing micro details, and ii) explores across different scales, assessing micro and macro information seamlessly. Additionally, based on the Mesorch architecture, the paper introduces two baseline models aimed at solving IML tasks through mesoscopic representation. Extensive experiments across four datasets have demonstrated that our models surpass the current state-of-the-art in terms of performance, computational complexity, and robustness.

Figures (6)

• Figure 1: Example of artifacts in three types of tampering. The red dashed box in the third column represents the range of the zoomed-in area in the fourth column. Red arrows in the fourth column point to artifacts that are considered tampering traces.
• Figure 2: Random samples from the CASIAv2 dataset. The red line marks the clear boundary of the tampered area. The first column shows tampering that is entirely unrelated to objects, while the other four columns show object-related tampering.
• Figure 3: Mesorch Framework: The input RGB image undergoes high- and low-frequency processing in the DCT Module to generate respective high-frequency and low-frequency representations. The Local Feature Module focuses on detecting fine-grained manipulation using both the original and high-frequency images, while the Global Feature Module captures object-level tampering cues by leveraging the original and low-frequency images. The Adaptive Weighting Module dynamically integrates these images by assigning pixel-level weights to local and global features. The final combined features are used for prediction and compared with ground-truth labels to compute the loss.
• Figure 4: Qualitative analysis of SOTA models. We randomly selected and compared four semantically manipulated images and one non-semantically manipulated image based on their respective proportions in the datasets. The first four images are semantically manipulated, while the last one is non-semantically manipulated.
• Figure 5: Ablation study on different backbones implemented in Mesorch Qualitatively "Conv" is ConvNeXt, "Res" is ResNet, and "Seg" is Segformer.