Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Generative AI Paradox: GenAI and the Erosion of Trust, the Corrosion of Information Verification, and the Demise of Truth

Emilio Ferrara

TL;DR

Generative AI enables synthetic realities that threaten shared epistemic ground and verification practices. The paper formalizes a layered stack of synthetic reality (content, identity, interaction, institutions), builds a taxonomy of harms across personal, economic, informational, and socio-technical dimensions, and analyzes qualitative shifts such as cost collapse, scale, customization, and provenance gaps. It ground these concepts with a 2023–2025 risk case bank spanning fraud, elections, harassment, and documentation, then advocates a defense-in-depth mitigation stack—provenance, governance, workflow redesign, and public resilience—tied to an agenda for measuring epistemic security. The authors conclude that ubiquitous synthetic content could drive societies to discount digital evidence, and they urge interdisciplinary research and measurement to build resilient verification ecosystems that preserve trust where it matters most.

Abstract

Generative AI (GenAI) now produces text, images, audio, and video that can be perceptually convincing at scale and at negligible marginal cost. While public debate often frames the associated harms as "deepfakes" or incremental extensions of misinformation and fraud, this view misses a broader socio-technical shift: GenAI enables synthetic realities; coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing. We argue that the most consequential risk is not merely the production of isolated synthetic artifacts, but the progressive erosion of shared epistemic ground and institutional verification practices as synthetic content, synthetic identity, and synthetic interaction become easy to generate and hard to audit. This paper (i) formalizes synthetic reality as a layered stack (content, identity, interaction, institutions), (ii) expands a taxonomy of GenAI harms spanning personal, economic, informational, and socio-technical risks, (iii) articulates the qualitative shifts introduced by GenAI (cost collapse, throughput, customization, micro-segmentation, provenance gaps, and trust erosion), and (iv) synthesizes recent risk realizations (2023-2025) into a compact case bank illustrating how these mechanisms manifest in fraud, elections, harassment, documentation, and supply-chain compromise. We then propose a mitigation stack that treats provenance infrastructure, platform governance, institutional workflow redesign, and public resilience as complementary rather than substitutable, and outline a research agenda focused on measuring epistemic security. We conclude with the Generative AI Paradox: as synthetic media becomes ubiquitous, societies may rationally discount digital evidence altogether.

The Generative AI Paradox: GenAI and the Erosion of Trust, the Corrosion of Information Verification, and the Demise of Truth

TL;DR

Generative AI enables synthetic realities that threaten shared epistemic ground and verification practices. The paper formalizes a layered stack of synthetic reality (content, identity, interaction, institutions), builds a taxonomy of harms across personal, economic, informational, and socio-technical dimensions, and analyzes qualitative shifts such as cost collapse, scale, customization, and provenance gaps. It ground these concepts with a 2023–2025 risk case bank spanning fraud, elections, harassment, and documentation, then advocates a defense-in-depth mitigation stack—provenance, governance, workflow redesign, and public resilience—tied to an agenda for measuring epistemic security. The authors conclude that ubiquitous synthetic content could drive societies to discount digital evidence, and they urge interdisciplinary research and measurement to build resilient verification ecosystems that preserve trust where it matters most.

Abstract

Generative AI (GenAI) now produces text, images, audio, and video that can be perceptually convincing at scale and at negligible marginal cost. While public debate often frames the associated harms as "deepfakes" or incremental extensions of misinformation and fraud, this view misses a broader socio-technical shift: GenAI enables synthetic realities; coherent, interactive, and potentially personalized information environments in which content, identity, and social interaction are jointly manufactured and mutually reinforcing. We argue that the most consequential risk is not merely the production of isolated synthetic artifacts, but the progressive erosion of shared epistemic ground and institutional verification practices as synthetic content, synthetic identity, and synthetic interaction become easy to generate and hard to audit. This paper (i) formalizes synthetic reality as a layered stack (content, identity, interaction, institutions), (ii) expands a taxonomy of GenAI harms spanning personal, economic, informational, and socio-technical risks, (iii) articulates the qualitative shifts introduced by GenAI (cost collapse, throughput, customization, micro-segmentation, provenance gaps, and trust erosion), and (iv) synthesizes recent risk realizations (2023-2025) into a compact case bank illustrating how these mechanisms manifest in fraud, elections, harassment, documentation, and supply-chain compromise. We then propose a mitigation stack that treats provenance infrastructure, platform governance, institutional workflow redesign, and public resilience as complementary rather than substitutable, and outline a research agenda focused on measuring epistemic security. We conclude with the Generative AI Paradox: as synthetic media becomes ubiquitous, societies may rationally discount digital evidence altogether.
Paper Structure (45 sections, 3 figures)

This paper contains 45 sections, 3 figures.

Table of Contents

  1. Introduction
  2. Contributions and scope
  3. A taxonomy of GenAI risks and harms
  4. Personal loss
  5. Financial and economic damage
  6. Information manipulation
  7. Socio-technical and infrastructural risks
  8. Epistemic and institutional integrity
  9. From synthetic media to synthetic reality
  10. Layer 1: Synthetic content
  11. Layer 2: Synthetic identity
  12. Layer 3: Synthetic interaction
  13. Layer 4: Synthetic institutions
  14. What you can't tell apart can harm you: why GenAI changes the game
  15. Cost collapse and commoditization
  16. ...and 30 more sections

Figures (3)

  • Figure 1: (Top Left) In January 2024, the r/StableDiffusion community on Reddit demonstrated a proof-of-concept workflow to synthetically generate personas and (Bottom Left) proofs of identity. (Top Right) GenAI can produce lifelike depictions of never-occurred events (MJv5 prompt: "president biden and supreme leader of iran shaking hands"). (Bottom Right) Subliminal messages in generated content (optical illusion reads OBEY).
  • Figure 2: Synthetic reality as a layered stack. Generative systems first produce synthetic content (text, image, audio, video), which enables synthetic identity (impersonation/persona fabrication) and synthetic interaction (adaptive, socially present dialogue). These layers can mutually reinforce one another, amplifying credibility and persuasion, while shifting verification burdens onto institutions (e.g., journalism, courts, elections, finance) as high-conviction artifacts become cheap and abundant. This structure illustrates how low-cost artifacts escalate into systemic epistemic failure.
  • Figure 3: A Research Agenda for Epistemic Security. Addressing synthetic reality risks requires a shift from measuring artifact authenticity to measuring systemic resilience. The agenda rests on four pillars: (1) operational metrics, (2) interactive benchmarks, (3) adversarial robustness, and (4) institutional redesign, all grounded in an analysis of equity and differential harm.