An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation and Dynamic Loss Weighting

TL;DR

This work tackles ownership protection for latent diffusion models by proposing EW-LoRA, a parameter-efficient watermarking method that injects low-rank LoRA adapters into the VAE decoder while keeping the base weights frozen. It couples this with a Dynamic Loss Weight Scheduler (DLWS) to rapidly balance watermark fidelity and image quality, achieving fast convergence with minimal parameter overhead. Empirical results show EW-LoRA attains high watermark bit accuracy (BitACC) and image fidelity (PSNR) across multiple datasets and base LDMs, often outperforming full-parameter or other APW approaches, and remains robust to post-processing, overwriting, and removal attacks. The approach generalizes well and offers a practical, scalable solution for watermarking large diffusion models, with potential applicability to broader generative architectures.

Abstract

The rapid proliferation of Deep Neural Networks (DNNs) is driving a surge in model watermarking technologies, as the trained models themselves constitute valuable intellectual property. Existing watermarking approaches primarily focus on modifying model parameters or altering sampling behaviors. However, with the emergence of increasingly large models, improving the efficiency of watermark embedding becomes essential to manage increasing computational demands. Prioritizing efficiency not only optimizes resource utilization, making the watermarking process more applicable for large models, but also mitigates potential degradation of model performance. In this paper, we propose an efficient watermarking method for Latent Diffusion Models (LDMs) based on Low-Rank Adaptation (LoRA). The core idea is to introduce trainable low-rank parameters into the frozen LDM to embed watermark, thereby preserving the integrity of the original model weights. Furthermore, a dynamic loss weight scheduler is designed to adaptively balance the objectives of generative quality and watermark fidelity, enabling the model to achieve effective watermark embedding with minimal impact on quality of the generated images. Experimental results show that the proposed method ensures fast and accurate watermark embedding and a high quality of the generated images, at the same time maintaining a level of robustness aligned - in some cases superior - with state-of-the-art approaches. Moreover, the method generalizes well across different datasets and base LDMs. Codes are available at: https://github.com/MrDongdongLin/EW-LoRA.

Figures (8)

• Figure 1: A comparative overview of watermarking pipelines for latent diffusion models.
• Figure 2: The working flowchart of EW-LoRA. In a latent diffusion model, the parameters in the image decoder are selected for introducing LoRA. The LoRA parameters $A$ and $B$ are trained with a pre-trained watermark decoder. During training, a dynamic loss weight scheduler is proposed to balance the generative task and the watermark embedding task.
• Figure 3: Watermarking performance for applying LoRA in different network blocks in the VAE decoder of an LDM.
• Figure 4: Watermarking performance for applying LoRA in single layer in the VAE decoder of an LDM.
• Figure 5: Comparison between images generated by original SD v1.4 and our EW-LoRA watermarked SD v1.4: the leftmost six images and the rightmost six are generated, respectively, without and with a text prompt.