Secure and Efficient Watermarking for Latent Diffusion Models in Model Distribution Scenarios

TL;DR

This work addresses watermarking for latent diffusion models in model distribution contexts where widespread access raises leakage and evasion risks. It introduces DistriMark, combining a watermark-network controller security mechanism with a secure latent-watermark distribution to enforce mandatory embedding and robust verification, while decoupling watermark injection from VAE fine-tuning to reduce overhead. Robustness is achieved through adversarial training and diffusion-inversion-aware verification, leading to superior performance against image-level and model-level attacks compared to six baselines, with strong detection and multi-bit traceability and competitive image quality. The framework offers practical, scalable copyright protection for distributed LDMs and sets the stage for further enhancements in adversarial robustness and deployment efficiency.

Abstract

Latent diffusion models have exhibited considerable potential in generative tasks. Watermarking is considered to be an alternative to safeguard the copyright of generative models and prevent their misuse. However, in the context of model distribution scenarios, the accessibility of models to large scale of model users brings new challenges to the security, efficiency and robustness of existing watermark solutions. To address these issues, we propose a secure and efficient watermarking solution. A new security mechanism is designed to prevent watermark leakage and watermark escape, which considers watermark randomness and watermark-model association as two constraints for mandatory watermark injection. To reduce the time cost of training the security module, watermark injection and the security mechanism are decoupled, ensuring that fine-tuning VAE only accomplishes the security mechanism without the burden of learning watermark patterns. A watermark distribution-based verification strategy is proposed to enhance the robustness against diverse attacks in the model distribution scenarios. Experimental results prove that our watermarking consistently outperforms existing six baselines on effectiveness and robustness against ten image processing attacks and adversarial attacks, while enhancing security in the distribution scenarios.

Figures (7)

• Figure 1: Watermark framework comparison with existing solutions.
• Figure 2: Framework of the proposed DistriMark watermarking scheme for Latent Diffusion Model.
• Figure 3: Generated image comparison under the security mechanism. Images in each sample from left to right are Watermarked Initial Latent Variables (WIL for short) without VAE-based fine-tuning (fine-tuning for short), WIL with fine-tuning, non-WIL without fine-tuning, and non-WIL with fine-tuning, representatively.
• Figure 4: Adversarial samples obtained from adaptive adversarial sample attack and reconstrctive attack.
• Figure 5: Results against adaptive adversarial sample attack (Left) and Reconstructive attack (Right).