SWIFT: Semantic Watermarking for Image Forgery Thwarting

TL;DR

SWIFT addresses image integrity under benign and malicious edits by embedding a semantic caption as a high-dimensional unit-norm vector $X \in \mathbb{R}^D$ with $D=256$ into the image using a modified HiDDeN-based encoder/decoder (Hide-$\mathbb{R}$). It decouples watermarking from encoding and uses a variable-rate TCCSK modulation layer to transmit a compressed caption, coupled with a confidence metric $\rho$ linked to the Message Recovery Rate for reliable authenticity decisions. The main contributions include the Hide-$\mathbb{R}$ watermarking layer, caption compression via arithmetic coding guided by an LLM (LLMZip), and a probabilistic decoding framework that provides actionable trust signals under transforms. Practically, SWIFT delivers robust semantic integrity verification suitable for moderation pipelines and watermark-enabled provenance, with demonstrated improvements over state-of-the-art methods across a range of benign and malignant edits.

Abstract

This paper proposes a novel approach towards image authentication and tampering detection by using watermarking as a communication channel for semantic information. We modify the HiDDeN deep-learning watermarking architecture to embed and extract high-dimensional real vectors representing image captions. Our method improves significantly robustness on both malign and benign edits. We also introduce a local confidence metric correlated with Message Recovery Rate, enhancing the method's practical applicability. This approach bridges the gap between traditional watermarking and passive forensic methods, offering a robust solution for image integrity verification.

Figures (6)

• Figure 1: Overview of SWIFT. To ensure the integrity of an image, Alice first leverages the message layer to tailor a semantic representation of the image, in our case a caption compressed in a lossless fashion. The resulting bit stream is fed to the TCCSK modulation layer thus enabling security and confidence (see \ref{['sec:decoding_confidence']}), and then to the watermarking layer based on our Hide-$\mathbb{R}$ encoder-decoder neural network. At reception, Bob executes the inverse process with a secret key and obtains both the caption and the p-value $\rho$. If $\rho$ is low enough, Bob can entrust the decoded caption and compare the image he received with the caption, enabling comparison between a proxy of the original image and the received one.
• Figure 2: Length distribution of BLIP2 captions encoded by OPT-125m version. We show that the finetuned version leads to entropy reduction and thus is more efficient to encode captions.
• Figure 3: Hide-$\mathbb{R}$ architecture. We use a L2 norm to control the watermark power to enforce a target PSNR on both the watermark signal in the embedder and on the decoded vector in the extractor.
• Figure 4: Performance of different encoding schemes under additive white Gaussian noise. $S_{256}^{\sqrt{3}}$ is the spherical lattice from spreadingvectors, capable of encoding $22108160 \simeq 2^{24.4}$ messages. TCCSK is most suited for our use case as it can cope with variable length messages.
• Figure 5: Representation of the message space after modulation. The circles $\mathcal{V}_{M},\mathcal{V}_{\hat{M}}$ illustrate Voronoï cells mapping to different binary messages on the surface of the hyper-sphere. $X$ is associated to the message to be hidden $M$. The extraction retrieves $Y$, which is decoded into $\hat{M}$, while $\hat{X}$ results from the modulation of $\hat{M}$, accounting for the perfect representation of $\hat{M}$. The similarity between $Y$ and $\hat{X}$ is given by $C$. This provides a confidence score for the decoding as explained in \ref{['sec:decoding_confidence']}. The given example illustrates a failure case with a wrong decoded message and a low confidence reflected by a high value of $\rho$.