Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning

Quan Zhang, Yuxin Qi, Xi Tang, Jinwei Fang, Xi Lin, Ke Zhang, Chun Yuan

TL;DR

This work addresses image manipulation detection by repurposing SAM through automated cross-view prompt learning. It introduces IMDPrompter, which leverages four views—RGB and semantic-agnostic SRM, Bayer, and Noiseprint—and employs Optimal Prompt Selection, Cross-View Prompt Consistency, Cross-view Feature Perception, and a Prompt Mixing Module to enable end-to-end manipulation localization. The training objective combines L_Seg-sam, L_Seg-p, L_CPC, and L_Img-level with weights λ1, λ2, λ3, and uses an Otsu-based adaptive threshold for image-level decisions, yielding strong IND and OOD performance across five datasets. Empirical results show IMDPrompter outperforms state-of-the-art methods on pixel- and image-level metrics, with notable robustness to distortions and significant gains on out-of-domain data. This framework demonstrates the potential of automated prompt learning for leveraging large pretrained segmentation models in forensic vision tasks and could extend to other foundational models beyond SAM.

Abstract

Using extensive training data from SA-1B, the Segment Anything Model (SAM) has demonstrated exceptional generalization and zero-shot capabilities, attracting widespread attention in areas such as medical image segmentation and remote sensing image segmentation. However, its performance in the field of image manipulation detection remains largely unexplored and unconfirmed. There are two main challenges in applying SAM to image manipulation detection: a) reliance on manual prompts, and b) the difficulty of single-view information in supporting cross-dataset generalization. To address these challenges, we develops a cross-view prompt learning paradigm called IMDPrompter based on SAM. Benefiting from the design of automated prompts, IMDPrompter no longer relies on manual guidance, enabling automated detection and localization. Additionally, we propose components such as Cross-view Feature Perception, Optimal Prompt Selection, and Cross-View Prompt Consistency, which facilitate cross-view perceptual learning and guide SAM to generate accurate masks. Extensive experimental results from five datasets (CASIA, Columbia, Coverage, IMD2020, and NIST16) validate the effectiveness of our proposed method.

IMDPrompter: Adapting SAM to Image Manipulation Detection by Cross-View Automated Prompt Learning

TL;DR

This work addresses image manipulation detection by repurposing SAM through automated cross-view prompt learning. It introduces IMDPrompter, which leverages four views—RGB and semantic-agnostic SRM, Bayer, and Noiseprint—and employs Optimal Prompt Selection, Cross-View Prompt Consistency, Cross-view Feature Perception, and a Prompt Mixing Module to enable end-to-end manipulation localization. The training objective combines L_Seg-sam, L_Seg-p, L_CPC, and L_Img-level with weights λ1, λ2, λ3, and uses an Otsu-based adaptive threshold for image-level decisions, yielding strong IND and OOD performance across five datasets. Empirical results show IMDPrompter outperforms state-of-the-art methods on pixel- and image-level metrics, with notable robustness to distortions and significant gains on out-of-domain data. This framework demonstrates the potential of automated prompt learning for leveraging large pretrained segmentation models in forensic vision tasks and could extend to other foundational models beyond SAM.

Abstract

Using extensive training data from SA-1B, the Segment Anything Model (SAM) has demonstrated exceptional generalization and zero-shot capabilities, attracting widespread attention in areas such as medical image segmentation and remote sensing image segmentation. However, its performance in the field of image manipulation detection remains largely unexplored and unconfirmed. There are two main challenges in applying SAM to image manipulation detection: a) reliance on manual prompts, and b) the difficulty of single-view information in supporting cross-dataset generalization. To address these challenges, we develops a cross-view prompt learning paradigm called IMDPrompter based on SAM. Benefiting from the design of automated prompts, IMDPrompter no longer relies on manual guidance, enabling automated detection and localization. Additionally, we propose components such as Cross-view Feature Perception, Optimal Prompt Selection, and Cross-View Prompt Consistency, which facilitate cross-view perceptual learning and guide SAM to generate accurate masks. Extensive experimental results from five datasets (CASIA, Columbia, Coverage, IMD2020, and NIST16) validate the effectiveness of our proposed method.

Paper Structure

This paper contains 19 sections, 19 equations, 10 figures, 18 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Proposed Method
  4. Multi-view feature representation
  5. Optimal Prompt Selection
  6. Cross-View Prompt Consistency
  7. Cross-view Feature Perception
  8. Prompt Mixing Module
  9. Training and Inference Pipeline
  10. Experiment
  11. Comparison with State-of-the-Art Method
  12. Ablation Studies
  13. Conslusion
  14. ACKNOWLEDGEMENTS
  15. Appendix
  16. ...and 4 more sections

Figures (10)

  • Figure 1: Improvements of proposed IMDPrompter over existing pipelines of baselines: (a). Automated prompt learning without the need for manual input. (b). Flexible integration of semantic-agnostic information crucial for Image Manipulation Detection.
  • Figure 2: Overall framework of IMDPrompter. The prompter part consists of four views: RGB, SRM, Bayar and Noiseprint. OPS selects the optimal prediction from the four views to generate the best prompt. CPC enhances cross-view consistency. CFP achieves cross-view feature perception fusion. PMM achieves a mixture of multiple prompt information.
  • Figure 3: Architecture of the Cross-view Feature Perception (CFP) unit.
  • Figure 4: Visualization Comparison of Image Manipulation Detection and Localization Results
  • Figure 5: Robustness Analysis Against JPEG Compression and Gaussian Blur Interference
  • ...and 5 more figures