AuthenLoRA: Entangling Stylization with Imperceptible Watermarks for Copyright-Secure LoRA Adapters

TL;DR

AuthenLoRA presents a joint optimization framework that embeds imperceptible, traceable watermarks into LoRA adapters used for diffusion-based stylization, tightly coupling watermarking with image generation. The method introduces a diagonal scaling mechanism, an expanded fine-tuning scope including ResNet blocks, and a two-stage training process with latent watermark pretraining and multi-objective loss to balance style fidelity with watermark reliability. A zero-message regularization strategy and perceptual residual suppression (PRTS) are proposed to minimize false positives and watermark residuals, while robustness is demonstrated against LoRA merging, pruning, and quantization, as well as cross-base-model and generation-parameter variations. The results show AuthenLoRA achieves high-fidelity stylization, near-perfect watermark extraction under benign conditions, and strong robustness to adversarial processing, at the cost of modestly larger model size and perceptible residuals compared with some post-hoc methods. Open-source implementation is provided to enable provenance tracking and copyright enforcement for stylized LoRA adapters.

Abstract

Low-Rank Adaptation (LoRA) offers an efficient paradigm for customizing diffusion models, but its ease of redistribution raises concerns over unauthorized use and the generation of untraceable content. Existing watermarking techniques either target base models or verify LoRA modules themselves, yet they fail to propagate watermarks to generated images, leaving a critical gap in traceability. Moreover, traceability watermarking designed for base models is not tightly coupled with stylization and often introduces visual degradation or high false-positive detection rates. To address these limitations, we propose AuthenLoRA, a unified watermarking framework that embeds imperceptible, traceable watermarks directly into the LoRA training process while preserving stylization quality. AuthenLoRA employs a dual-objective optimization strategy that jointly learns the target style distribution and the watermark-induced distribution shift, ensuring that any image generated with the watermarked LoRA reliably carries the watermark. We further design an expanded LoRA architecture for enhanced multi-scale adaptation and introduce a zero-message regularization mechanism that substantially reduces false positives during watermark verification. Extensive experiments demonstrate that AuthenLoRA achieves high-fidelity stylization, robust watermark propagation, and significantly lower false-positive rates compared with existing approaches. Open-source implementation is available at: https://github.com/ShiFangming0823/AuthenLoRA

Figures (8)

• Figure 1: Visual results of stylization targets, images generated by stylization-only LoRA and watermarked images generated by AuthenLoRA. Our proposed AuthenLoRA preserves visual quality nearly indistinguishable from that of stylization-only LoRA-generated images.
• Figure 2: The threat model. The LoRA trainer trains and owns the copyright of AuthenLoRA. Users use AuthenLoRA to generate stylized images with watermarks indicating their identity.
• Figure 3: The overall framework of AuthenLoRA. Our proposed training pipeline consists of two stages. (a) The first stage is latent watermark paradigm pre-training. In this stage, we jointly train the watermark encoder $E_s$ and decoder $D_s$ in the latent space. (b) In the second stage, we jointly train LoRA and the watermark mapper. LoRA will learn both the pre-trained watermark paradigm and the stylized content. The watermark mapper is used to embed bit-level watermark message into LoRA.
• Figure 4: The watermarked images and their corresponding residuals generated by watermark encoder trained without the PRTS loss. Regions exhibiting high-saturation artifacts are highlighted. The residual intensity has been amplified 5x to enhance visibility.
• Figure 5: Effect of generation parameters ((a) Classifier Free Guidance, (b) Sample Steps and (c) Sampler) on bit accuracy and TPR@FPR=$10^{-6}$ during image generation for Stable Diffusion.