Adoption of Watermarking for Generative AI Systems in Practice and Implications under the new EU AI Act

TL;DR

The paper investigates how watermarking and deep-fake labeling can be implemented in practice under the EU AI Act, combining legal analysis with an empirical survey of 50 AI image systems. It identifies four deployment scenarios to map responsibilities and highlights ambiguities in Article 50 and related provisions. Empirically, it finds limited adoption of machine-readable markings (38%) and deep-fake labels (18%), with metadata embeddings and C2PA-based fingerprints being most common among marks, and visible disclosures remaining rare. The authors discuss policy and technical implications, advocate system- or model-level integration to improve compliance, and provide public tooling to detect watermarks in images to support enforcement and research.

Abstract

AI-generated images have become so good in recent years that individuals often cannot distinguish them any more from "real" images. This development, combined with the rapid spread of AI-generated content online, creates a series of societal risks. Watermarking, a technique that involves embedding information within images and other content to indicate their AI-generated nature, has emerged as a primary mechanism to address the risks posed by AI-generated content. Indeed, watermarking and AI labelling measures are now becoming a legal requirement in many jurisdictions, including under the 2024 European Union AI Act. Despite the widespread use of AI image generation systems, the practical implications and the current status of implementation of these measures remain largely unexamined. The present paper therefore provides both an empirical and a legal analysis of these measures. In our legal analysis, we identify four categories of generative AI deployment scenarios and outline how the legal obligations could apply in each category. In our empirical analysis, we find that only a minority number of AI image generators currently implement adequate watermarking (38%) and deep fake labelling (18%) practices. In response, we suggest a range of avenues of how the implementation of these legally mandated techniques can be improved, and publicly share our tooling for the detection of watermarks in images.

Figures (7)

• Figure 1: The applicability of the AI Act’s (water)marking and disclosure rules along a simplified version of the generative AI supply chain.
• Figure 2: Example of Hugging Face’s Inference widget (bottom right), as found on the popular stable-diffusion XL-1.0 model page. This widget allows end-users to directly generate images from a model; it is yet unclear who in this case bears responsibility for implementing the AI Act’s transparency requirements.
• Figure 3: Results from analysing the 50 generative AI image systems, using the metrics from Table \ref{['tab:metrics']}
• Figure 4: Overview of the different machine-readable marking solutions found in the 50 AI systems analysed.
• Figure 5: Examples of visible markings applied in the content generated with the neutral prompt ('A PhD student')