Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploring the Use of Abusive Generative AI Models on Civitai

Yiluo Wei, Yiming Zhu, Pan Hui, Gareth Tyson

TL;DR

This study provides the first large-scale empirical analysis of abusive content on an AIGC social platform, Civitai, using a dataset of 87,042 models and 2,740,149 images collected via API and web scraping. By combining GPT-assisted thematic annotation, real-person name extraction, and OpenAI moderation, it characterizes six content themes, highlights the prevalence of NSFW content and deepfakes, and demonstrates that NSFW content attracts higher engagement while deepfakes show distinct, celebrity-targeted patterns. The work also reveals that abusive creators occupy central positions in the platform’s follower network, suggesting brokerage and influence that can amplify reach and impact. Findings underscore the need for enhanced moderation, prompt-based safeguards, and network-aware interventions to mitigate abuse on AIGC communities, with implications for policy and platform design across similar ecosystems.

Abstract

The rise of generative AI is transforming the landscape of digital imagery, and exerting a significant influence on online creative communities. This has led to the emergence of AI-Generated Content (AIGC) social platforms, such as Civitai. These distinctive social platforms allow users to build and share their own generative AI models, thereby enhancing the potential for more diverse artistic expression. Designed in the vein of social networks, they also provide artists with the means to showcase their creations (generated from the models), engage in discussions, and obtain feedback, thus nurturing a sense of community. Yet, this openness also raises concerns about the abuse of such platforms, e.g., using models to disseminate deceptive deepfakes or infringe upon copyrights. To explore this, we conduct the first comprehensive empirical study of an AIGC social platform, focusing on its use for generating abusive content. As an exemplar, we construct a comprehensive dataset covering Civitai, the largest available AIGC social platform. Based on this dataset of 87K models and 2M images, we explore the characteristics of content and discuss strategies for moderation to better govern these platforms.

Exploring the Use of Abusive Generative AI Models on Civitai

TL;DR

This study provides the first large-scale empirical analysis of abusive content on an AIGC social platform, Civitai, using a dataset of 87,042 models and 2,740,149 images collected via API and web scraping. By combining GPT-assisted thematic annotation, real-person name extraction, and OpenAI moderation, it characterizes six content themes, highlights the prevalence of NSFW content and deepfakes, and demonstrates that NSFW content attracts higher engagement while deepfakes show distinct, celebrity-targeted patterns. The work also reveals that abusive creators occupy central positions in the platform’s follower network, suggesting brokerage and influence that can amplify reach and impact. Findings underscore the need for enhanced moderation, prompt-based safeguards, and network-aware interventions to mitigate abuse on AIGC communities, with implications for policy and platform design across similar ecosystems.

Abstract

The rise of generative AI is transforming the landscape of digital imagery, and exerting a significant influence on online creative communities. This has led to the emergence of AI-Generated Content (AIGC) social platforms, such as Civitai. These distinctive social platforms allow users to build and share their own generative AI models, thereby enhancing the potential for more diverse artistic expression. Designed in the vein of social networks, they also provide artists with the means to showcase their creations (generated from the models), engage in discussions, and obtain feedback, thus nurturing a sense of community. Yet, this openness also raises concerns about the abuse of such platforms, e.g., using models to disseminate deceptive deepfakes or infringe upon copyrights. To explore this, we conduct the first comprehensive empirical study of an AIGC social platform, focusing on its use for generating abusive content. As an exemplar, we construct a comprehensive dataset covering Civitai, the largest available AIGC social platform. Based on this dataset of 87K models and 2M images, we explore the characteristics of content and discuss strategies for moderation to better govern these platforms.
Paper Structure (17 sections, 6 figures, 4 tables)

This paper contains 17 sections, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Primer on Civitai
  3. Data Collection Methodology
  4. Data Collection
  5. Data Augmentation
  6. Baseline Prompt Datasets
  7. Exploring Model and Image Themes (RQ1)
  8. Overview of Themes
  9. Overview of Creators' Themes
  10. Relationships between Themes
  11. Exploring The Creation of Abusive Images (RQ2)
  12. Exploring user engagement with Abusive Models and Images (RQ3)
  13. Exploring The Network Positions of Abusive Creators (RQ4)
  14. Social Network Definition
  15. Centrality Analysis
  16. ...and 2 more sections

Figures (6)

  • Figure 1: CDF of the percentage of models and images, by the creator that contain a specific theme.
  • Figure 2: Heat maps of phi coefficient showing pair-wise category correlation between each of the six themes.
  • Figure 3: Comparison of the distribution of productivity among distinct types of models, measured as the number of images per model.
  • Figure 4: Distribution of prompts' NSFW score. Prompt with a degree exceeding the threshold (0.53) will be reported as NSFW prompt by OpenAI's moderation API.
  • Figure 5: Comparison of the distribution of prompts' NSFW score between our Civitai dataset and other two selected prompt datasets.
  • ...and 1 more figures