Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Durability: Benchmark Insights into Multimodal Watermarking

Jielin Qiu, William Han, Xuandong Zhao, Shangbang Long, Christos Faloutsos, Lei Li

TL;DR

This work addresses the robustness of multimodal watermarks embedded in content generated by image and text models. It introduces a cross-modal benchmark thatinjects 4 image watermarks and 4 text watermarks into 5,000 image-caption and 5,000 caption-generated images across 100 image perturbations and 63 text perturbations, evaluated over 16 diverse benchmark models. The study finds that watermark robustness is highly sensitive to distribution shifts, with perturbation type driving performance (e.g., Zoom Blur and OCR errors being particularly disruptive) and model choice shaping outcomes (e.g., SDXL-Lightning excels for image robustness, while LLaVA fares better for text). By systematically contrasting watermarking strategies and perturbation categories, the work provides actionable guidance for developing more robust multimodal watermarking techniques and offers a public codebase and benchmark for future research.

Abstract

With the development of large models, watermarks are increasingly employed to assert copyright, verify authenticity, or monitor content distribution. As applications become more multimodal, the utility of watermarking techniques becomes even more critical. The effectiveness and reliability of these watermarks largely depend on their robustness to various disturbances. However, the robustness of these watermarks in real-world scenarios, particularly under perturbations and corruption, is not well understood. To highlight the significance of robustness in watermarking techniques, our study evaluated the robustness of watermarked content generated by image and text generation models against common real-world image corruptions and text perturbations. Our results could pave the way for the development of more robust watermarking techniques in the future. Our project website can be found at \url{https://mmwatermark-robustness.github.io/}.

Evaluating Durability: Benchmark Insights into Multimodal Watermarking

TL;DR

This work addresses the robustness of multimodal watermarks embedded in content generated by image and text models. It introduces a cross-modal benchmark thatinjects 4 image watermarks and 4 text watermarks into 5,000 image-caption and 5,000 caption-generated images across 100 image perturbations and 63 text perturbations, evaluated over 16 diverse benchmark models. The study finds that watermark robustness is highly sensitive to distribution shifts, with perturbation type driving performance (e.g., Zoom Blur and OCR errors being particularly disruptive) and model choice shaping outcomes (e.g., SDXL-Lightning excels for image robustness, while LLaVA fares better for text). By systematically contrasting watermarking strategies and perturbation categories, the work provides actionable guidance for developing more robust multimodal watermarking techniques and offers a public codebase and benchmark for future research.

Abstract

With the development of large models, watermarks are increasingly employed to assert copyright, verify authenticity, or monitor content distribution. As applications become more multimodal, the utility of watermarking techniques becomes even more critical. The effectiveness and reliability of these watermarks largely depend on their robustness to various disturbances. However, the robustness of these watermarks in real-world scenarios, particularly under perturbations and corruption, is not well understood. To highlight the significance of robustness in watermarking techniques, our study evaluated the robustness of watermarked content generated by image and text generation models against common real-world image corruptions and text perturbations. Our results could pave the way for the development of more robust watermarking techniques in the future. Our project website can be found at \url{https://mmwatermark-robustness.github.io/}.
Paper Structure (41 sections, 7 figures, 20 tables)

This paper contains 41 sections, 7 figures, 20 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text Watermarking
  4. Image Watermarking
  5. Robustness of Image or Text Watermarks
  6. Robustness of Multimodal Models
  7. Preliminary: Invisible Watermark Detection
  8. Perturbation Methods
  9. Image Perturbation.
  10. Text Perturbation.
  11. Watermarks
  12. Image Watermarks (Image-WM)
  13. Text Watermarks (Text-WM)
  14. Experiments
  15. Experimental Settings
  16. ...and 26 more sections

Figures (7)

  • Figure 1: The overall pipeline of our watermarking robustness study. We add watermarks to the generated content and evaluate their robustness under image corruptions and text perturbations.
  • Figure 2: Performance comparison of different models under image perturbations.
  • Figure 3: Performance comparison of different models under text perturbations.
  • Figure 4: Comparisons of different [Top] image corruption and [Bottom] text perturbation methods using Stable Diffusion and LLaVA, respectively. All the results have been averaged on different severity levels.
  • Figure 5: Model comparisons under [Top] image corrections and [Bottom] text perturbations. All the results have been averaged on the performance under all image/text perturbations.
  • ...and 2 more figures

Theorems & Definitions (2)

  • Definition 3.1: Invisible watermark
  • Definition 3.2: Watermark detection