Have LLMs Reopened the Pandora's Box of AI-Generated Fake News?

TL;DR

The paper investigates whether LLMs reopen the Pandora's box of AI-generated fake news by running a university-scale two-phase competition that analyzes both generation and detection, including human versus LLM performance and the role of visuals. It finds that while LLMs are adept at identifying real news, both humans and LLMs struggle with AI-generated fake content, especially in collaborative human–AI generation scenarios. The work also uncovers topic- and prompt-dependent variability in detection, with visuals offering only modest gains and processing mode influencing outcomes. These findings underscore the need for robust, multi-modal, topic-aware detection strategies and careful governance of AI-assisted content creation.

Abstract

With the rise of AI-generated content spewed at scale from large language models (LLMs), genuine concerns about the spread of fake news have intensified. The perceived ability of LLMs to produce convincing fake news at scale poses new challenges for both human and automated fake news detection systems. To address this gap, this paper presents the findings from a university-level competition that aimed to explore how LLMs can be used by humans to create fake news, and to assess the ability of human annotators and AI models to detect it. A total of 110 participants used LLMs to create 252 unique fake news stories, and 84 annotators participated in the detection tasks. Our findings indicate that LLMs are ~68% more effective at detecting real news than humans. However, for fake news detection, the performance of LLMs and humans remains comparable (~60% accuracy). Additionally, we examine the impact of visual elements (e.g., pictures) in news on the accuracy of detecting fake news stories. Finally, we also examine various strategies used by fake news creators to enhance the credibility of their AI-generated content. This work highlights the increasing complexity of detecting AI-generated fake news, particularly in collaborative human-AI settings.

Figures (6)

• Figure 1: Box-plot comparison of correctly identified real, fake, and total stories by humans and GPT-4o.
• Figure 2: Density plots comparing the performance of humans and GPT-4o models (batch and single modes) across four metrics. Top-left: Precision; Top-right: Recall; Bottom-left: False Positive Rate (FPR); Bottom-right: False Negative Rate (FNR).
• Figure A1: Co-usage heatmap of primary prompting strategies (P1-P4) and secondary output optimization strategies (S1-S7). Note: Multiple primary and secondary strategies can be applied simultaneously. For simplicity, we only map the relationships between primary and secondary strategies.
• Figure A2: Scatter plot displaying the frequency of occurrence of terms in the indicators generated by GPT-4o ($Temp_{best}$=0.5), with the x-axis representing terms more frequently associated with news incorrectly identified as real by GPT-4o, and the y-axis representing terms more frequently associated with news correctly identified as fake by GPT-4o.
• Figure A3: Scatter plot displaying the frequency of occurrence of terms in the indicators generated by Gemini ($Temp_{best}$=0.3), with the x-axis representing terms more frequently associated with news incorrectly identified as real by Gemini, and the y-axis representing terms more frequently associated with news correctly identified as fake by Gemini.