Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking VLMs for Image Forgery Detection and Localization

Shaofeng Guo, Jiequan Cui, Richang Hong

Abstract

With the rapid rise of Artificial Intelligence Generated Content (AIGC), image manipulation has become increasingly accessible, posing significant challenges for image forgery detection and localization (IFDL). In this paper, we study how to fully leverage vision-language models (VLMs) to assist the IFDL task. In particular, we observe that priors from VLMs hardly benefit the detection and localization performance and even have negative effects due to their inherent biases toward semantic plausibility rather than authenticity. Additionally, the location masks explicitly encode the forgery concepts, which can serve as extra priors for VLMs to ease their training optimization, thus enhancing the interpretability of detection and localization results. Building on these findings, we propose a new IFDL pipeline named IFDL-VLM. To demonstrate the effectiveness of our method, we conduct experiments on 9 popular benchmarks and assess the model performance under both in-domain and cross-dataset generalization settings. The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.Code is available at: https://github.com/sha0fengGuo/IFDL-VLM.

Rethinking VLMs for Image Forgery Detection and Localization

Abstract

With the rapid rise of Artificial Intelligence Generated Content (AIGC), image manipulation has become increasingly accessible, posing significant challenges for image forgery detection and localization (IFDL). In this paper, we study how to fully leverage vision-language models (VLMs) to assist the IFDL task. In particular, we observe that priors from VLMs hardly benefit the detection and localization performance and even have negative effects due to their inherent biases toward semantic plausibility rather than authenticity. Additionally, the location masks explicitly encode the forgery concepts, which can serve as extra priors for VLMs to ease their training optimization, thus enhancing the interpretability of detection and localization results. Building on these findings, we propose a new IFDL pipeline named IFDL-VLM. To demonstrate the effectiveness of our method, we conduct experiments on 9 popular benchmarks and assess the model performance under both in-domain and cross-dataset generalization settings. The experimental results show that we consistently achieve new state-of-the-art performance in detection, localization, and interpretability.Code is available at: https://github.com/sha0fengGuo/IFDL-VLM.
Paper Structure (22 sections, 4 equations, 22 figures, 10 tables)

This paper contains 22 sections, 4 equations, 22 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method --- IFDL-VLM
  4. Do Vision-Language Alignment Priors Benefit IFDL Performance?
  5. IFDL-VLM
  6. Experiments
  7. Experimental Setup
  8. Detection&Localization Comparisons
  9. Evaluation on SID-Set
  10. Evaluation on Cross-Dataset Generalization
  11. Evaluation on Language Interpretability
  12. Conclusion
  13. Appendix / Supplemental Material
  14. Ablation on $\alpha$ for Region-aware Visual Feature Enhancement
  15. Effect of Unfreezing CLIP
  16. ...and 7 more sections

Figures (22)

  • Figure 1: Comparison with the existing IFDL pipeline. (a) Existing IFDL algorithms enhance the interpretability with language explanations from VLMs. Meanwhile, the $<SEG>$ token and the $<CLS>$ token for localization and detection are derived from the responses of VLMs. (b) Our IFDL method decouples the optimization of language explanations and detection&localization. Moreover, the detection&localization results are used as extra inputs to assist the optimization of language explanation with VLMs.
  • Figure 2: VLMs exhibit a bias toward semantic plausibility rather than authenticity. As a result, they are often insensitive to semantically consistent or subtle forgeries, such as the replacement of a cat instance in (a) or the addition of a new person in (b). This inherent bias ultimately hinders IFDL performance.
  • Figure 3: Our IFDL-VLM framework. IFDL-VLM method decouples the optimization of detection&localization from the language explanation generation. (a) Detection&localization expert training with VIT and SAM. (b) Region-aware visual feature enhancement technique for language explanation generation.
  • Figure 4: Qualitative comparison on localization mask and language explanations.
  • Figure 5: Overall preference distribution in human evaluation.
  • ...and 17 more figures