SAPL: Semantic-Agnostic Prompt Learning in CLIP for Weakly Supervised Image Manipulation Localization

TL;DR

SAPL tackles weakly supervised image manipulation localization by steering CLIP away from high-level semantics and toward boundary artifacts. It introduces semantic-agnostic prompt learning with two synergistic components: ECPL, which infuses edge cues into learnable text prompts, and HECL, which enforces hierarchical edge representations via contrastive learning on edge patches. This combination yields sharper localization heatmaps and strong cross-dataset generalization, achieving state-of-the-art results on multiple benchmarks without pixel-level annotations. The approach offers practical impact by reducing labeling costs while delivering precise manipulation localization across diverse datasets and perturbations.

Abstract

Malicious image manipulation threatens public safety and requires efficient localization methods. Existing approaches depend on costly pixel-level annotations which make training expensive. Existing weakly supervised methods rely only on image-level binary labels and focus on global classification, often overlooking local edge cues that are critical for precise localization. We observe that feature variations at manipulated boundaries are substantially larger than in interior regions. To address this gap, we propose Semantic-Agnostic Prompt Learning (SAPL) in CLIP, which learns text prompts that intentionally encode non-semantic, boundary-centric cues so that CLIPs multimodal similarity highlights manipulation edges rather than high-level object semantics. SAPL combines two complementary modules Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL) to exploit edge information in both textual and visual spaces. The proposed ECPL leverages edge-enhanced image features to generate learnable textual prompts via an attention mechanism, embedding semantic-irrelevant information into text features, to guide CLIP focusing on manipulation edges. The proposed HECL extract genuine and manipulated edge patches, and utilize contrastive learning to boost the discrimination between genuine edge patches and manipulated edge patches. Finally, we predict the manipulated regions from the similarity map after processing. Extensive experiments on multiple public benchmarks demonstrate that SAPL significantly outperforms existing approaches, achieving state-of-the-art localization performance.

Figures (8)

• Figure 1: Subtle manipulation artifacts can be observed along the boundaries of manipulated regions. Our approach produces sharper heatmaps around manipulation boundaries.
• Figure 2: Four low-level statistics (mean local variance, mean gradient magnitude, intensity skewness and excess kurtosis) computed over four region types (manipulated edge / manipulated inner / authentic edge / authentic inner) on CASIAv2. Manipulated edges consistently show higher noise and gradient and altered higher-order statistics (mean ± std), supporting the motivation to focus on boundary cues.
• Figure 3: The SAPL architecture. SAPL implements semantic-agnostic prompt learning via two complementary modules: Edge-aware Contextual Prompt Learning (ECPL) and Hierarchical Edge Contrastive Learning (HECL). During training, SAPL extracts patch features and soft edge maps, applies ECPL for prompt generation and HECL for edge contrastive learning, and optimizes classification plus contrastive losses. At inference, it uses patch–text similarity to produce localization maps and decide the class.
• Figure 4: Canny-based Adaptive Soft Edge Map.
• Figure 5: Hierarchical Edge Contrastive Learning. The ECL block computes similarity maps between patch features and text features, then uses the soft edge map to select the high-confidence patches for contrastive learning.