Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Industrialized Deception: The Collateral Effects of LLM-Generated Misinformation on Digital Ecosystems

Alexander Loth, Martin Kappes, Marc-Oliver Pahl

TL;DR

Industrialized deception through LLM-generated misinformation threatens digital ecosystems by scaling deceptive content. The paper shifts from survey to practical countermeasures, introducing JudgeGPT and RogueGPT as an end-to-end pipeline for studying human perception and misinformation detection. It documents an evolving threat landscape, including multimodal content, agentic AI, and the Generative AI Paradox, and discusses mitigation strategies ranging from LLM-based detection to provenance standards and prebunking. The work provides a foundation for epistemic-security-oriented research and platform-level resilience.

Abstract

Generative AI and misinformation research has evolved since our 2024 survey. This paper presents an updated perspective, transitioning from literature review to practical countermeasures. We report on changes in the threat landscape, including improved AI-generated content through Large Language Models (LLMs) and multimodal systems. Central to this work are our practical contributions: JudgeGPT, a platform for evaluating human perception of AI-generated news, and RogueGPT, a controlled stimulus generation engine for research. Together, these tools form an experimental pipeline for studying how humans perceive and detect AI-generated misinformation. Our findings show that detection capabilities have improved, but the competition between generation and detection continues. We discuss mitigation strategies including LLM-based detection, inoculation approaches, and the dual-use nature of generative AI. This work contributes to research addressing the adverse impacts of AI on information quality.

Industrialized Deception: The Collateral Effects of LLM-Generated Misinformation on Digital Ecosystems

TL;DR

Industrialized deception through LLM-generated misinformation threatens digital ecosystems by scaling deceptive content. The paper shifts from survey to practical countermeasures, introducing JudgeGPT and RogueGPT as an end-to-end pipeline for studying human perception and misinformation detection. It documents an evolving threat landscape, including multimodal content, agentic AI, and the Generative AI Paradox, and discusses mitigation strategies ranging from LLM-based detection to provenance standards and prebunking. The work provides a foundation for epistemic-security-oriented research and platform-level resilience.

Abstract

Generative AI and misinformation research has evolved since our 2024 survey. This paper presents an updated perspective, transitioning from literature review to practical countermeasures. We report on changes in the threat landscape, including improved AI-generated content through Large Language Models (LLMs) and multimodal systems. Central to this work are our practical contributions: JudgeGPT, a platform for evaluating human perception of AI-generated news, and RogueGPT, a controlled stimulus generation engine for research. Together, these tools form an experimental pipeline for studying how humans perceive and detect AI-generated misinformation. Our findings show that detection capabilities have improved, but the competition between generation and detection continues. We discuss mitigation strategies including LLM-based detection, inoculation approaches, and the dual-use nature of generative AI. This work contributes to research addressing the adverse impacts of AI on information quality.
Paper Structure (30 sections, 2 figures)

This paper contains 30 sections, 2 figures.

Table of Contents

  1. Introduction
  2. The Domain
  3. Evolution Since 2024
  4. Structural Overview
  5. Digital Ecosystems and Information Integrity
  6. Functioning of Generative AI for Fake News Generation
  7. Technical Background
  8. Generative Artificial Intelligence
  9. Generative Adversarial Networks (GANs)
  10. Variational Autoencoders (VAEs)
  11. Transformer Models
  12. BERT
  13. GPT
  14. Overview of Recent Developments
  15. The Agentic Shift in Misinformation
  16. ...and 15 more sections

Figures (2)

  • Figure 1: Structural overview of Generative AI's impact on Fake News. The dual-use nature (purple dashed arrow) illustrates how the same technologies enable both creation and detection of synthetic content.
  • Figure 2: GPT architecture: tokens are embedded with positional encoding, processed through $N$ transformer blocks (self-attention $\rightarrow$ layer norm $\rightarrow$ FFN $\rightarrow$ layer norm, with residual connections at each stage), then projected to output vocabulary.