WAVES: Benchmarking the Robustness of Image Watermarks

TL;DR

WAVES addresses the lack of standardized evaluation for image watermarks by proposing a comprehensive benchmark that jointly assesses detection and user identification under 26 stress tests (distortions, regenerations, and adversarial embeddings) across three datasets. It introduces a unified evaluation workflow with eight image-quality metrics and Performance-vs-Quality plots, centering on $TPR@0.1%FPR$ to quantify detection robustness. Applying WAVES to three prominent watermarks (Stable Signature, Tree-Ring, StegaStamp) reveals novel vulnerabilities, such as susceptibility to regeneration and surrogate-detector attacks, and highlights the risks of using publicly available VAEs. The framework serves as a practical tool for researchers and industry to systematically diagnose weaknesses and guide the development of more robust watermarking approaches for AI-generated imagery.

Abstract

In the burgeoning age of generative AI, watermarks act as identifiers of provenance and artificial content. We present WAVES (Watermark Analysis Via Enhanced Stress-testing), a benchmark for assessing image watermark robustness, overcoming the limitations of current evaluation methods. WAVES integrates detection and identification tasks and establishes a standardized evaluation protocol comprised of a diverse range of stress tests. The attacks in WAVES range from traditional image distortions to advanced, novel variations of diffusive, and adversarial attacks. Our evaluation examines two pivotal dimensions: the degree of image quality degradation and the efficacy of watermark detection after attacks. Our novel, comprehensive evaluation reveals previously undetected vulnerabilities of several modern watermarking algorithms. We envision WAVES as a toolkit for the future development of robust watermarks. The project is available at https://wavesbench.github.io/

Figures (32)

• Figure 1: WAVES establishes a standardized evaluation framework that encompasses a comprehensive suite of stress tests including both existing and newly proposed stronger attacks (denoted by $^*$).
• Figure 2: Evaluation workflow.
• Figure 3: Regeneration attacks on Tree-Ringk. Regen-Diff is a single diffusive regeneration and Rinse-[N]xDiff is a rinsing one with $N$ repeated diffusions, with the number of noising steps as attack strength. Regen-VAE uses a pre-trained VAE with quality factor as strength and Regen-KLVAE uses pre-trained KL-VAEs with bottleneck size as strength. RinseD-VAE applies a VAE as a denoiser after Rinse-4xDiff.
• Figure 4: Adversarial embedding attacks target Tree-Ring at strengths of {2/255, 4/255, 6/255, 8/255}. Tree-Ring shows vulnerability to embedding attacks, especially when the adversary can access the VAE being used.
• Figure 5: Three settings for training the surrogate detector. The Generator is the victim generator under attack. We externalize the watermarking process for simplicity, but it could be in-processing watermarks. After training the surrogate detectors, the adversary performs PGD attacks on them to flip the labels.