VideoMarkBench: Benchmarking Robustness of Video Watermarking

TL;DR

VideoMarkBench introduces the first systematic benchmark for evaluating robustness of video watermarking against removal and forgery perturbations in AI-generated videos, combining the VideoMarkData AI-generated dataset with a real Kinetics-400 set, four watermarking methods, seven aggregation strategies, and 12 perturbation types across white-box, black-box, and no-box threat models. The study shows that while watermarking is accurate with no perturbations, all evaluated methods are vulnerable to adversarial attacks, with aggregation strategy choice influencing robustness and logit-level aggregation often outperforming BA-based schemes. It also demonstrates varying robustness across generative models and video styles and provides insights into practical defense strategies and evaluation protocols for future research. Overall, the work provides a rigorous, scalable framework for assessing watermarking robustness and highlights the urgent need for more resilient techniques in real-world, perturbation-rich environments.

Abstract

The rapid development of video generative models has led to a surge in highly realistic synthetic videos, raising ethical concerns related to disinformation and copyright infringement. Recently, video watermarking has been proposed as a mitigation strategy by embedding invisible marks into AI-generated videos to enable subsequent detection. However, the robustness of existing video watermarking methods against both common and adversarial perturbations remains underexplored. In this work, we introduce VideoMarkBench, the first systematic benchmark designed to evaluate the robustness of video watermarks under watermark removal and watermark forgery attacks. Our study encompasses a unified dataset generated by three state-of-the-art video generative models, across three video styles, incorporating four watermarking methods and seven aggregation strategies used during detection. We comprehensively evaluate 12 types of perturbations under white-box, black-box, and no-box threat models. Our findings reveal significant vulnerabilities in current watermarking approaches and highlight the urgent need for more robust solutions. Our code is available at https://github.com/zhengyuan-jiang/VideoMarkBench.

Figures (23)

• Figure 1: Summary of our VideoMarkBench.
• Figure 2: White-box watermark removal results in the first scenario.
• Figure 3: White-box watermark forgery results in the first scenario.
• Figure 4: White-box attack results in the second scenario with different aggregation strategies.
• Figure 5: Square Attack watermark removal results. Perturbations are $l_{\infty}$ bounded by 0.05.