Deepfakes on Demand: the rise of accessible non-consensual deepfake image generators
Will Hawkins, Chris Russell, Brent Mittelstadt
TL;DR
This study addresses the rise of AI-generated deepfakes by empirically analyzing publicly downloadable T2I model variants on two major platforms, focusing on Stable Diffusion and Flux. It employs a three-part methodology (platform-level analysis on Civitai, detailed model-level analysis across SD and Flux, and assessment of Terms of Service and accessibility) to quantify prevalence, usage patterns, and governance gaps. The findings reveal tens of thousands of deepfake variants, millions of downloads, and a strong bias toward female-targeted content, with LoRA fine-tuning enabling rapid, low-resource creation and distribution. The results underscore urgent needs for proactive platform enforcement, provenance and watermarking, and regulatory interventions to mitigate NCII risks and the spread of photorealistic deepfakes.
Abstract
Advances in multimodal machine learning have made text-to-image (T2I) models increasingly accessible and popular. However, T2I models introduce risks such as the generation of non-consensual depictions of identifiable individuals, otherwise known as deepfakes. This paper presents an empirical study exploring the accessibility of deepfake model variants online. Through a metadata analysis of thousands of publicly downloadable model variants on two popular repositories, Hugging Face and Civitai, we demonstrate a huge rise in easily accessible deepfake models. Almost 35,000 examples of publicly downloadable deepfake model variants are identified, primarily hosted on Civitai. These deepfake models have been downloaded almost 15 million times since November 2022, with the models targeting a range of individuals from global celebrities to Instagram users with under 10,000 followers. Both Stable Diffusion and Flux models are used for the creation of deepfake models, with 96% of these targeting women and many signalling intent to generate non-consensual intimate imagery (NCII). Deepfake model variants are often created via the parameter-efficient fine-tuning technique known as low rank adaptation (LoRA), requiring as few as 20 images, 24GB VRAM, and 15 minutes of time, making this process widely accessible via consumer-grade computers. Despite these models violating the Terms of Service of hosting platforms, and regulation seeking to prevent dissemination, these results emphasise the pressing need for greater action to be taken against the creation of deepfakes and NCII.