Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking

TL;DR

This work tackles the unsafe and costly limitations of low-entropy watermarking in LLM-generated text by introducing Invisible Entropy (IE), a lightweight, entropy-aware watermarking framework that does not require access to the original LLM during detection. IE comprises a Unified Feature Extractor to unify representations across tokenizers, an Entropy Tagger to predict low-entropy next-token events, and a Threshold Navigator to adaptively set entropy thresholds for robust, natural watermarks. Empirically, IE achieves up to a 99% reduction in detection-time parameters while delivering state-of-the-art or comparable watermarking performance on HumanEval and MBPP, with strong robustness to paraphrasing and generalization to other watermarking schemes via the Threshold Navigator. The work advances practical deployment of watermarking for safe AI by reducing leakage risk, speeding up detection, and maintaining text quality and detectability in low-entropy contexts.

Abstract

Logit-based LLM watermarking traces and verifies AI-generated content by maintaining green and red token lists and increasing the likelihood of green tokens during generation. However, it fails in low-entropy scenarios, where predictable outputs make green token selection difficult without disrupting natural text flow. Existing approaches address this by assuming access to the original LLM to calculate entropy and selectively watermark high-entropy tokens. However, these methods face two major challenges: (1) high computational costs and detection delays due to reliance on the original LLM, and (2) potential risks of model leakage. To address these limitations, we propose Invisible Entropy (IE), a watermarking paradigm designed to enhance both safety and efficiency. Instead of relying on the original LLM, IE introduces a lightweight feature extractor and an entropy tagger to predict whether the entropy of the next token is high or low. Furthermore, based on theoretical analysis, we develop a threshold navigator that adaptively sets entropy thresholds. It identifies a threshold where the watermark ratio decreases as the green token count increases, enhancing the naturalness of the watermarked text and improving detection robustness. Experiments on HumanEval and MBPP datasets demonstrate that IE reduces parameter size by 99\% while achieving performance on par with state-of-the-art methods. Our work introduces a safe and efficient paradigm for low-entropy watermarking. https://github.com/Carol-gutianle/IE https://huggingface.co/datasets/Carol0110/IE-Tagger

Figures (7)

• Figure 1: Existing watermarking methods in low-entropy scenarios face safety and cost challenges, while our method addresses them efficiently and securely.
• Figure 2: Overview of IE (Invisible Entropy). The model includes three components: the Unified Feature Extractor for tokenizer compatibility and feature extraction, the Entropy Tagger to predict if the next token’s entropy exceeds threshold $\tau$, and the Threshold Navigator to optimize $\tau$ for effective watermarking, naturalness, and robustness. Tokens are color-coded as red (red list), green (green list), and gray (unwatermarked). This example shows the search stopping at $\tau = 0.6$. At $\tau = 0.9$, insufficient watermarking occurs, while at $\tau = 0.3$, excessive low-entropy classification causes token generation issues (e.g., the underscore "_").
• Figure 3: Analysis of the Entropy Tagger. (a) Comparison of applying the Entropy Tagger at different stages: generation-detection versus detection-only. (b) The relationship between Entropy Tagger accuracy and its effectiveness in watermarking. (c) Demonstration of the superior performance of the Entropy Tagger compared to a surrogate model and randomly set entropy.
• Figure 4: (a) Type-I Error probability and the distribution of detection statistic $z$ for human-written text. (b) Impact of threshold navigator search directions. (c) Robustness of detection to paraphrasing attacks.
• Figure 5: Effectiveness of the Threshold Navigator. (a) Improved detectability and quality with the Navigator across $\delta$. (b) Improved UES with the Navigator. (c) Generalizability to SWEET: Pass@1 vs. AUROC, demonstrating similar improvements. (d) UES comparison for SWEET, showing significant gains with the Navigator.