SoK: On the Offensive Potential of AI

TL;DR

This SoK provides a comprehensive, multi-source systematization of offensive AI (OAI) by examining 95 academic papers, 38 InfoSec briefings, a layperson survey (N=$N=549$), and input from 12 experts. It introduces an OAI Assessment Checklist to map use-cases, targets, and cost/benefit across diverse sources, and it offers an online tool to extend analyses to future works. The study reveals that OAI is heterogeneous, affecting systems, humans, and society, with many techniques evaluated against toy systems and few robust countermeasures considered, especially in the economic dimension. It also shows that industrial briefings highlight risks and real-world applicability that academia often overlooks, and it provides a forward-looking research agenda (open problems and concerns) to guide defenses, governance, and interdisciplinary collaboration. Overall, the work underscores the need for ongoing, transparent, and cross-domain monitoring of offensive AI to inform effective mitigation strategies.

Abstract

Our society increasingly benefits from Artificial Intelligence (AI). Unfortunately, more and more evidence shows that AI is also used for offensive purposes. Prior works have revealed various examples of use cases in which the deployment of AI can lead to violation of security and privacy objectives. No extant work, however, has been able to draw a holistic picture of the offensive potential of AI. In this SoK paper we seek to lay the ground for a systematic analysis of the heterogeneous capabilities of offensive AI. In particular we (i) account for AI risks to both humans and systems while (ii) consolidating and distilling knowledge from academic literature, expert opinions, industrial venues, as well as laypeople -- all of which being valuable sources of information on offensive AI. To enable alignment of such diverse sources of knowledge, we devise a common set of criteria reflecting essential technological factors related to offensive AI. With the help of such criteria, we systematically analyze: 95 research papers; 38 InfoSec briefings (from, e.g., BlackHat); the responses of a user study (N=549) entailing individuals with diverse backgrounds and expertise; and the opinion of 12 experts. Our contributions not only reveal concerning ways (some of which overlooked by prior work) in which AI can be offensively used today, but also represent a foothold to address this threat in the years to come.

Figures (17)

• Figure 1: Targets and knowledge sources related to offensive AI.
• Figure 2: Systematic Literature Review.We collect over 3000 papers from various repositories. After filtering, screening and inter-researcher discussions, we coalesce 95 papers on OAI which we consider in this SoK.
• Figure 3: Questionnaire.Depending on the answers, participants have to respond to up to four broad questions (e.g., no specific time frame is given). Some questions expect open answers, ensuring freedom to share any concern.
• Figure 4: OAI use cases not covered by MITRE ATT&CK (technical papers).Some focus on society and privacy (OSN=online social networks).
• Figure 5: Quantitative results (laypeople).Sankey chart of the closed questions.