ForgeryGPT: Multimodal Large Language Model For Explainable Image Forgery Detection and Localization

TL;DR

ForgeryGPT tackles the lack of explainability in image forgery detection by integrating a Mask-Aware Forgery Extractor with a multimodal LLM, enabling pixel-level localization and interactive, explainable reasoning. It introduces a three-stage training regimen (Image-Text Alignment, Mask-Text Alignment, Task-Specific Instruction Tuning) and dedicated data pipelines to fuse vision-language understanding with forgery-specific knowledge. Empirical results show state-of-the-art localization and detection across diverse datasets, robustness to distortions, and compelling human-evaluated explanations. The work demonstrates the feasibility of endowing MLLMs with high-level forensic semantics for trustworthy, interactive image integrity analysis in real-world scenarios.

Abstract

Multimodal Large Language Models (MLLMs), such as GPT4o, have shown strong capabilities in visual reasoning and explanation generation. However, despite these strengths, they face significant challenges in the increasingly critical task of Image Forgery Detection and Localization (IFDL). Moreover, existing IFDL methods are typically limited to the learning of low-level semantic-agnostic clues and merely provide a single outcome judgment. To tackle these issues, we propose ForgeryGPT, a novel framework that advances the IFDL task by capturing high-order forensics knowledge correlations of forged images from diverse linguistic feature spaces, while enabling explainable generation and interactive dialogue through a newly customized Large Language Model (LLM) architecture. Specifically, ForgeryGPT enhances traditional LLMs by integrating the Mask-Aware Forgery Extractor, which enables the excavating of precise forgery mask information from input images and facilitating pixel-level understanding of tampering artifacts. The Mask-Aware Forgery Extractor consists of a Forgery Localization Expert (FL-Expert) and a Mask Encoder, where the FL-Expert is augmented with an Object-agnostic Forgery Prompt and a Vocabulary-enhanced Vision Encoder, allowing for effectively capturing of multi-scale fine-grained forgery details. To enhance its performance, we implement a three-stage training strategy, supported by our designed Mask-Text Alignment and IFDL Task-Specific Instruction Tuning datasets, which align vision-language modalities and improve forgery detection and instruction-following capabilities. Extensive experiments demonstrate the effectiveness of the proposed method.

Figures (14)

• Figure 1: Comparison between our ForgeryGPT and existing methods. "Forgery Mask" denotes the ground truth mask of the forged image. Current IFDL methods only provide a forgery score without offering any explainability, with the forgery detection capability heavily reliant on threshold settings. Existing MLLMs either lack forgery detection capabilities entirely or fail to provide accurate explainability for detected forgeries. In contrast, ForgeryGPT not only detects and locates manipulation traces correctly but also accurately identifies forged objects, determines the forgery type, and offers a detailed reasoning process.
• Figure 2: Overview of the proposed ForgeryGPT. The left panel shows the overall architecture, which comprises an Image Encoder, a Mask-Aware Forgery Extractor, and a Large Language Model. The right panel provides a detailed view of the Mask-aware Forgery Extractor and the the Forgery Localization Expert.
• Figure 3: Overview of FL-Expert. It consists of the Object-agnostic Forgery Prompt module, frozen CLIP text and vision encoders, a Vocabulary-enhanced Vision Encoder, Multi-layer Attention Fusion, and a U-Net-shaped Decoder.
• Figure 4: Two training data generation pipelines: one for Mask-Text Alignment Pre-training, which creates caption data from forgery datasets, and another for Task-Specific Instruction Tuning, generating Q&A pairs for IFDL task.
• Figure 5: Our splicing method for constructing multi-granularity splicing manipulation sub-dataset, which uses detailed, multi-granularity segmentation masks to improve image manipulation, compared to traditional manual method and existing generative technique that rely on simpler segmentation.