LoRAGuard: An Effective Black-box Watermarking Approach for LoRAs

TL;DR

LoRAGuard introduces a robust black-box watermarking framework for LoRAs that remains detectable when LoRAs are added or negated, addressing the challenge of multiple LoRAs. The Yin-Yang watermark combines complementary components activated by addition and negation, while a shadow-model training regime improves reliability in multitask LoRA compositions. The method achieves near-perfect watermark verification with minimal impact on downstream task performance and shows strong robustness to fine-tuning and pruning, with practical stealth against certain defenses. This work enables practical traceability and IP protection for LoRA-based customization in both LLMs and diffusion models, with potential for broad adoption in open-source ecosystems.

Abstract

LoRA (Low-Rank Adaptation) has achieved remarkable success in the parameter-efficient fine-tuning of large models. The trained LoRA matrix can be integrated with the base model through addition or negation operation to improve performance on downstream tasks. However, the unauthorized use of LoRAs to generate harmful content highlights the need for effective mechanisms to trace their usage. A natural solution is to embed watermarks into LoRAs to detect unauthorized misuse. However, existing methods struggle when multiple LoRAs are combined or negation operation is applied, as these can significantly degrade watermark performance. In this paper, we introduce LoRAGuard, a novel black-box watermarking technique for detecting unauthorized misuse of LoRAs. To support both addition and negation operations, we propose the Yin-Yang watermark technique, where the Yin watermark is verified during negation operation and the Yang watermark during addition operation. Additionally, we propose a shadow-model-based watermark training approach that significantly improves effectiveness in scenarios involving multiple integrated LoRAs. Extensive experiments on both language and diffusion models show that LoRAGuard achieves nearly 100% watermark verification success and demonstrates strong effectiveness.

Figures (14)

• Figure 1: Watermark injection using BadNets: main task performance and watermark verification success rate under Addition and Negation with varying number of LoRAs.
• Figure 2: The overview of LoRAGuard. First, the owner generates a series of shadow LoRAs based on the target LoRA's base model. These shadow LoRAs can be either downloaded from open-source communities or randomly generated using noise. Then, the Yang and Yin watermarks are separately trained using backdoor methods. Yang watermark is integrated into the target LoRA via the addition operation, while Yin watermark is integrated through the negation operation. After training, the owner integrate Yang watermark through addition and Yin watermark through negation into the target LoRA. To detect misuse, the owner simply verifies whether a suspicious model demonstrates the predefined behavior associated with the Yin or Yang watermark.
• Figure 3: Image styles of Yin-Yang watermark.
• Figure 4: Main task performance and generated images before (a, b) and after (c, d) clip with Yang watermark triggered.
• Figure 5: The Number of LoRAs.