A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Model

TL;DR

VISA-Mark addresses the conflict between watermark detectability and visual-grounding fidelity in LVLMs by introducing a vision-aligned watermarking framework. It leverages a lightweight offline prefix-tuning extractor to produce Visual Evidence Weights, which guide an uncertainty-aware vocabulary partitioning and an evidence-calibrated logit perturbation to concentrate watermark strength on visually supported tokens. The approach maintains high detectability while substantially improving visual fidelity and text quality, supported by extensive experiments and ablations across multiple LVLMs and benchmarks. This work establishes a practical, reliability-preserving paradigm for multimodal watermarking in large vision-language systems.

Abstract

Watermarking has emerged as a pivotal solution for content traceability and intellectual property protection in Large Vision-Language Models (LVLMs). However, vision-agnostic watermarks introduce visually irrelevant tokens and disrupt visual grounding by enforcing indiscriminate pseudo-random biases, while some semantic-aware methods incur prohibitive inference latency due to rejection sampling. In this paper, we propose the VIsual Semantic Adaptive Watermark (VISA-Mark), a novel framework that embeds detectable signals while strictly preserving visual fidelity. Our approach employs a lightweight, efficiently trained prefix-tuner to extract dynamic Visual-Evidence Weights, which quantify the evidentiary support for candidate tokens based on the visual input. These weights guide an adaptive vocabulary partitioning and logits perturbation mechanism, concentrating watermark strength specifically on visually-supported tokens. By actively aligning the watermark with visual evidence, VISA-Mark effectively maintains visual fidelity. Empirical results confirm that VISA-Mark outperforms conventional methods with a 7.8% improvement in visual consistency (Chair-I) and superior semantic fidelity. The framework maintains highly competitive detection accuracy (96.88% AUC) and robust attack resilience (99.3%) without sacrificing inference efficiency, effectively establishing a new standard for reliability-preserving multimodal watermarking.

Figures (6)

• Figure 1: Paradigm comparison between our VISA-Mark and currently existing vocabulary partitioning-based watermark & semantic-aware watermark.
• Figure 2: Overview of VISA-Mark framework, which consists of three components: ($A$) Visual Evidence Extractor: A lightweight prefix-tuner trained offline through dense image-caption pairs ($A_1$), is deployed at inference time to extract Visual Evidence Weights ($A_2$). ($B$) Uncertainty-based Vocabulary Partitioning: Leverages logits entropy and the extracted weights to adaptively swap high-evidence tokens into the green-list, protecting visual fidelity. ($C$) Evidence-Calibrated Logit Perturbation: Applies a perturbation bias that scales with the Visual Evidence Weight and entropy, concentrating watermark strength on visually-grounded tokens.
• Figure 3: Text quality and visual consistency analysis between VISA-Mark and baseline methods. Left: Violin plots of perplexity scores; VISA-M shows a lower median and tighter distribution, indicating higher fluency. Middle: BERTScore versus token length; our method mitigates semantic degradation in long-text generation. Right: Chair-I versus token length; VISA-M maintains the lowest hallucination rate as generation grows, confirming robust visual fidelity.
• Figure 4: ROC curves evaluating detection performance under no-attack and three text attack scenarios (Word-Insert, Word-Delete, Synonym-Substitute at 5% rate). VISA-Mark (blue curve) demonstrates superior robustness, maintaining near-perfect AUC across all attacks, whereas baselines like DiP and Unbiased exhibit performance collapse.
• Figure 5: Training loss dynamics of the prefix-tuner on LLaVA-v1.5 (Left) and Qwen3-VL (Right) backbones over 2,438 steps (3 epochs). The light green lines represent the raw step-wise KL divergence loss, while the dark green lines depict the smoothed loss trajectory. Both models demonstrate rapid convergence in the early stages and maintain stability, validating the efficiency of our visual-evidence extraction learning.