Is Multilingual LLM Watermarking Truly Multilingual? A Simple Back-Translation Solution

TL;DR

The paper shows that existing multilingual watermarking methods fail to generalize across translation, especially for medium- and low-resource languages, due to semantic clustering limitations tied to tokenizer coverage. It introduces STEAM, a back-translation-based, model-agnostic defence that boosts watermark detectability by evaluating back-translations across multiple languages and choosing the strongest signal, with per-language $z$-score normalization to control distribution shifts. Across 17 languages and diverse attacks, STEAM substantially outperforms prior semantic-clustering approaches, achieving higher $AUC$ and $TPR@1\%$, and demonstrating robustness to language exposure and translator differences. The results advocate for fairness-focused design in watermarking and offer a practical, readily extensible technique to ensure watermark reliability across linguistic diversity. Overall, STEAM provides a scalable, non-invasive path to more reliable watermarking in a globally diverse linguistic landscape.

Abstract

Multilingual watermarking aims to make large language model (LLM) outputs traceable across languages, yet current methods still fall short. Despite claims of cross-lingual robustness, they are evaluated only on high-resource languages. We show that existing multilingual watermarking methods are not truly multilingual: they fail to remain robust under translation attacks in medium- and low-resource languages. We trace this failure to semantic clustering, which fails when the tokenizer vocabulary contains too few full-word tokens for a given language. To address this, we introduce STEAM, a back-translation-based detection method that restores watermark strength lost through translation. STEAM is compatible with any watermarking method, robust across different tokenizers and languages, non-invasive, and easily extendable to new languages. With average gains of +0.19 AUC and +40%p TPR@1% on 17 languages, STEAM provides a simple and robust path toward fairer watermarking across diverse languages.

Figures (6)

• Figure 1: (\ref{['fig:teaser-translation-attack']}) Our goal is to evaluate the robustness of LLM watermarks against translation attacks. (\ref{['fig:teaser-auc-drop']}) Our analysis reveals that existing multilingual watermarks fail to generalize across languages, while our approach (Steam) performs consistently better across a wide range of languages overlooked by previous work.
• Figure 2: Languages with larger tokenizer vocabularies have higher watermark robustness. Average AUC per language and model across three seeds. Lines are least squared regressions.
• Figure 3: Overview of Steam. A suspect text (Bengali) is back-translated into multiple supported languages to generate a pool of candidate texts. Each candidate, including the suspect text, is evaluated using a standard watermark detector to compute per-language $z$-score. Steam outputs the highest value across normalized and original $z$-scores.
• Figure 4: Tokenizer vocabulary favours high-resource languages. Percentage of words in multilingual dictionaries that appear in the tokenizer vocabulary.
• Figure 5: Tokenization of low-resource languages creates highly concentrated sub-character tokens. Percentage of top 10 tokens for Bengali (\ref{['fig:bn_aya']}) and for Tamil (\ref{['fig:ta_aya']}) using Aya-23 8B.