Perpetuating Misogyny with Generative AI: How Model Personalization Normalizes Gendered Harm

TL;DR

The paper investigates how personalized generative AI within open-source hubs like CivitAI can perpetuate gendered harm, including non-consensual deepfakes and misogynistic content. It adopts a multi-level analysis of image data, adapters, and training-tagging pipelines, revealing rising NSFW trends, gender biases in checkpoints, and widespread use of Danbooru-based captions in adapters. The findings highlight how platform dynamics, tagging ecosystems, and community norms contribute to entrenched patterns of sexualization and exploitation, and they discuss governance and technical safeguards as pathways to mitigation. The work underscores the need for proactive, cross-stakeholder interventions to curb downstream harm and promote consent and accountability in open-source AI ecosystems.

Abstract

Open-source text-to-image (TTI) pipelines have become dominant in the landscape of AI-generated visual content, driven by technological advances that enable users to personalize models through adapters tailored to specific tasks. While personalization methods such as LoRA offer unprecedented creative opportunities, they also facilitate harmful practices, including the generation of non-consensual deepfakes and the amplification of misogynistic or hypersexualized content. This study presents an exploratory sociotechnical analysis of CivitAI, the most active platform for sharing and developing open-source TTI models. Drawing on a dataset of more than 40 million user-generated images and over 230,000 models, we find a disproportionate rise in not-safe-for-work (NSFW) content and a significant number of models intended to mimic real individuals. We also observe a strong influence of internet subcultures on the tools and practices shaping model personalizations and resulting visual media. In response to these findings, we contextualize the emergence of exploitative visual media through feminist and constructivist perspectives on technology, emphasizing how design choices and community dynamics shape platform outcomes. Building on this analysis, we propose interventions aimed at mitigating downstream harm, including improved content moderation, rethinking tool design, and establishing clearer platform policies to promote accountability and consent.

Figures (14)

• Figure 1: Random sample of images taken from CivitAI in March 2024 with different NSFW levels as assigned by the platform.
• Figure 2: Schematic overview of the CivitAI platform, showing layered relationships between the curated front-end images, the underlying generative models, and the broader sociotechnical ecosystem. This includes tools, data sources, and cultural influences, that shape model development.
• Figure 3: Monthly distribution of generated images categorized by the NSFW browsing level assigned by CivitAI.
• Figure 4: Inferred age and gender predictions using MiVOLO on a 0.1% sample of temporally representative images from the 40 million image dataset.
• Figure 5: Co-occurrence network of the 300 most frequent promotional tags across all assets shared on CivitAI with the top three nodes indicating total occurrence rates and opacity and thickness intensity of co-occurrence rates.