AGMark: Attention-Guided Dynamic Watermarking for Large Vision-Language Models

TL;DR

This work tackles IP protection for LVLMs by addressing the fidelity-detectability trade-off that plagues existing watermarks. It introduces AGMark, a two-stage framework that dynamically identifies semantic-critical tokens at each decoding step using attention-guided visual evidence and contextual coherence, then adaptively partitions the vocabulary by jointly considering token entropy and weight density. The method achieves near-perfect detectability with $ ext{AUC} \ge 99.36\%$ and strong robustness ($\text{AUC} \ge 88.61\%$ under attacks) while delivering improvements in visual fidelity (CHAIR) and language quality, with only modest latency overhead. These results are demonstrated across three LVLMs on AMBER and MS-COCO, supported by ablations and attack analyses, establishing a practical, reliability-preserving watermarking paradigm for cross-modal generation. The core contributions include a dynamic semantic-weight extraction mechanism, an entropy-and-density-driven adaptive partitioning strategy, and extensive empirical validation of superior detectability, fidelity, and robustness in LVLM watermarking. Key formulas include the fusion of vision and context signals into a semantic critical score $\psi_t(k)$, the entropy-based uncertainty measure $\mathcal{H}_t^{norm}$, and the adaptive token ratio $\eta_t$, all of which underpin the adaptive watermark injection without compromising generation quality. $

Abstract

Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in Large Vision-Language Models (LVLMs). However, vision-agnostic watermarks may introduce visually irrelevant tokens and disrupt visual grounding by enforcing indiscriminate pseudo-random biases. Additionally, current vision-specific watermarks rely on a static, one-time estimation of vision critical weights and ignore the weight distribution density when determining the proportion of protected tokens. This design fails to account for dynamic changes in visual dependence during generation and may introduce low-quality tokens in the long tail. To address these challenges, we propose Attention-Guided Dynamic Watermarking (AGMark), a novel framework that embeds detectable signals while strictly preserving visual fidelity. At each decoding step, AGMark first dynamically identifies semantic-critical evidence based on attention weights for visual relevance, together with context-aware coherence cues, resulting in a more adaptive and well-calibrated evidence-weight distribution. It then determines the proportion of semantic-critical tokens by jointly considering uncertainty awareness (token entropy) and evidence calibration (weight density), thereby enabling adaptive vocabulary partitioning to avoid irrelevant tokens. Empirical results confirm that AGMark outperforms conventional methods, observably improving generation quality and yielding particularly strong gains in visual semantic fidelity in the later stages of generation. The framework maintains highly competitive detection accuracy (at least 99.36\% AUC) and robust attack resilience (at least 88.61\% AUC) without sacrificing inference efficiency, effectively establishing a new standard for reliability-preserving multi-modal watermarking.

Figures (5)

• Figure 1: Paradigm comparison of our AGMark, vision-agnostic watermarking methods, and recent LVLM watermarking methods.
• Figure 2: Overview of AGMark framework, which consists of two components: (A) Semantic Critical Weight Extracting: Dynamically extracting weight via similarity calculation to fuse vision critical weights and context critical weights. (B) Adaptive Vocabulary Partitioning: Leverages logits entropy and density of extracted weights to adaptively swap high evidence tokens into the green-list, protecting visual fidelity.
• Figure 3: Violin plots show the sample-wise performance distribution including Perplexity (Left), BertScore (Middle), and CHAIR (Right). For Perplexity and BertScore, the overlaid box plots indicate the median. For CHAIR, however, the box plot shows the mean, since the large number of zero values for AGMark and IE would otherwise yield a median of 0.
• Figure 4: Line charts depict length-wise performance retention across response token lengths ranging from 60 to 200 tokens, evaluated using three metrics: Perplexity (Left), BertScore (Mddle), and CHAIR (Right).
• Figure 5: AUC matrix for six watermarking methods under various attack scenarios, with AUC values reported in parentheses. Smaller differences between pre-attack and post-attack AUC indicate stronger robustness against the corresponding attacks. The relative performance drop comparison as shown in Appendix \ref{['apx:Relative Performance Drop']}.