Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

Nahema Marchal, Rachel Xu, Rasmi Elasmar, Iason Gabriel, Beth Goldberg, William Isaac

TL;DR

This study builds a two-part taxonomy of GenAI misuse by integrating a literature review with a qualitative analysis of roughly 191–200 real-world incidents from January 2023 to March 2024, spanning text, image, audio, and video modalities. It distinguishes exploitation of GenAI capabilities from attacks on GenAI systems, identifying ten capability-exploitation tactics and eight (or nine) system-compromise tactics, with Impersonation, Appropriated Likeness, Sockpuppeting, and NCII emerging as dominant. The empirical findings reveal that most misuse leverages readily accessible capabilities to manipulate opinions, monetize content, or commit fraud, rather than sophisticated model attacks, while technical and social mitigations are both necessary. The authors discuss governance and safety implications, including prebunking and detection approaches, to curb rapid growth in AI-enabled deception and manipulation across domains. The work informs policy makers, safety teams, and researchers about current threat patterns and guides future evaluation and intervention strategies amid accelerating GenAI capabilities.

Abstract

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.

Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data

TL;DR

This study builds a two-part taxonomy of GenAI misuse by integrating a literature review with a qualitative analysis of roughly 191–200 real-world incidents from January 2023 to March 2024, spanning text, image, audio, and video modalities. It distinguishes exploitation of GenAI capabilities from attacks on GenAI systems, identifying ten capability-exploitation tactics and eight (or nine) system-compromise tactics, with Impersonation, Appropriated Likeness, Sockpuppeting, and NCII emerging as dominant. The empirical findings reveal that most misuse leverages readily accessible capabilities to manipulate opinions, monetize content, or commit fraud, rather than sophisticated model attacks, while technical and social mitigations are both necessary. The authors discuss governance and safety implications, including prebunking and detection approaches, to curb rapid growth in AI-enabled deception and manipulation across domains. The work informs policy makers, safety teams, and researchers about current threat patterns and guides future evaluation and intervention strategies amid accelerating GenAI capabilities.

Abstract

Generative, multimodal artificial intelligence (GenAI) offers transformative potential across industries, but its misuse poses significant risks. Prior research has shed light on the potential of advanced AI systems to be exploited for malicious purposes. However, we still lack a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm. In this paper, we present a taxonomy of GenAI misuse tactics, informed by existing academic literature and a qualitative analysis of approximately 200 observed incidents of misuse reported between January 2023 and March 2024. Through this analysis, we illuminate key and novel patterns in misuse during this time period, including potential motivations, strategies, and how attackers leverage and abuse system capabilities across modalities (e.g. image, text, audio, video) in the wild.
Paper Structure (18 sections, 2 figures, 8 tables)

This paper contains 18 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Taxonomy of Generative AI Misuse Tactics
  4. Exploitation of GenAI capabilities
  5. Compromise of GenAI systems
  6. Findings
  7. Prevalence and modalities of misuse tactics
  8. Goals and strategies of misuse
  9. Opinion Manipulation
  10. Monetization & Profit
  11. Scam & Fraud
  12. Harassment
  13. Reach
  14. Discussion
  15. Limitations and further research
  16. ...and 3 more sections

Figures (2)

  • Figure 1: Frequency of tactics across categories.
  • Figure 2: Top strategies associated with each misuse goal.