Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Characterizing AI-Generated Misinformation on Social Media

Chiara Drolsbach, Nicolas Pröllochs

TL;DR

This study addresses the prevalence and diffusion of AI-generated misinformation on X by analyzing 91,452 crowd-annotated posts flagged via Community Notes. It combines GPT-4-turbo-based identification with LLM-driven annotation to compare AI-generated versus traditional misinformation across content, authorship, virality, and perceived harm/believability. Key findings show AI-generated posts are more entertainment- and media-focused, originate from larger yet more insular accounts, spread more virally, and are only slightly less believable or harmful. The work highlights the need for platform defenses that account for AI-generated content as a distinct factor in misinformation dynamics and informs future research on detection, policy, and media literacy.

Abstract

AI-generated misinformation (e.g., deepfakes) poses a growing threat to information integrity on social media. However, prior research has largely focused on its potential societal consequences rather than its real-world prevalence. In this study, we conduct a large-scale empirical analysis of AI-generated misinformation on the social media platform X. Specifically, we analyze a dataset comprising N=91,452 misleading posts, both AI-generated and non-AI-generated, that have been identified and flagged through X's Community Notes platform. Our analysis yields four main findings: (i) AI-generated misinformation is more often centered on entertaining content and tends to exhibit a more positive sentiment than conventional forms of misinformation, (ii) it is more likely to originate from smaller user accounts, (iii) despite this, it is significantly more likely to go viral, and (iv) it is slightly less believable and harmful compared to conventional misinformation. Altogether, our findings highlight the unique characteristics of AI-generated misinformation on social media. We discuss important implications for platforms and future research.

Characterizing AI-Generated Misinformation on Social Media

TL;DR

This study addresses the prevalence and diffusion of AI-generated misinformation on X by analyzing 91,452 crowd-annotated posts flagged via Community Notes. It combines GPT-4-turbo-based identification with LLM-driven annotation to compare AI-generated versus traditional misinformation across content, authorship, virality, and perceived harm/believability. Key findings show AI-generated posts are more entertainment- and media-focused, originate from larger yet more insular accounts, spread more virally, and are only slightly less believable or harmful. The work highlights the need for platform defenses that account for AI-generated content as a distinct factor in misinformation dynamics and informs future research on detection, policy, and media literacy.

Abstract

AI-generated misinformation (e.g., deepfakes) poses a growing threat to information integrity on social media. However, prior research has largely focused on its potential societal consequences rather than its real-world prevalence. In this study, we conduct a large-scale empirical analysis of AI-generated misinformation on the social media platform X. Specifically, we analyze a dataset comprising N=91,452 misleading posts, both AI-generated and non-AI-generated, that have been identified and flagged through X's Community Notes platform. Our analysis yields four main findings: (i) AI-generated misinformation is more often centered on entertaining content and tends to exhibit a more positive sentiment than conventional forms of misinformation, (ii) it is more likely to originate from smaller user accounts, (iii) despite this, it is significantly more likely to go viral, and (iv) it is slightly less believable and harmful compared to conventional misinformation. Altogether, our findings highlight the unique characteristics of AI-generated misinformation on social media. We discuss important implications for platforms and future research.
Paper Structure (19 sections, 1 equation, 8 figures, 1 table)

This paper contains 19 sections, 1 equation, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Data and Methods
  4. Data source
  5. Identification of AI-generated misinformation
  6. Annotation of post characteristics
  7. Empirical Analysis
  8. Content characteristics (RQ1)
  9. Author characteristics (RQ2)
  10. Virality (RQ3)
  11. Harmfulness & believability (RQ4)
  12. Discussion
  13. Implications
  14. Limitations & future research
  15. Conclusion
  16. ...and 4 more sections

Figures (8)

  • Figure 1: Examples of posts on X that shares a deepfake in form of an (a) Image, (b) Video, or (c) no media (here: URL to media content) and the corresponding Community Notes.
  • Figure 2: Distribution of media types/modalities in AI-generated vs. non-AI-generated misleading posts. $N=$ 91452.0.
  • Figure 3: Distribution of sentiment in AI-generated vs. non-AI-generated misleading posts. $N=$ 3000.0.
  • Figure 4: Distribution of topics in AI-generated vs. non-AI-generated misleading posts. $N=$ 3000.0.
  • Figure 5: Complementary cumulative distribution functions (CCDFs) for AI vs. non-AI generated misleading posts, shown separately for follower counts (a), followee counts (b), account age (c), and the number of posts per author (d). $N=$ 91452.0.
  • ...and 3 more figures