Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Alexander Loth, Martin Kappes, Marc-Oliver Pahl

Abstract

Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News

Abstract

Can humans tell whether a news article was written by a person or a large language model (LLM)? We investigate this question using JudgeGPT, a study platform that independently measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales. From 2,318 judgments collected from 1,054 participants across content generated by six LLMs, we report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text (p > .05, Welch's t-test); (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; (3) self-reported domain expertise predicts judgment accuracy (r = .35, p < .001) whereas political orientation does not (r = -.10, n.s.); (4) clustering reveals distinct response strategies ("Skeptics" vs. "Believers"); and (5) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue. The answer, in short, is no: humans cannot reliably tell. These results indicate that user-side detection is not a viable defense and motivate system-level countermeasures such as cryptographic content provenance.

Paper Structure

This paper contains 16 sections, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Dual-Axis Assessment.
  5. Stimulus Design.
  6. Data Collection.
  7. Findings
  8. Finding 1: Machine-Generated Text Is Not Detectable by Participants.
  9. Finding 2: Detection Failure Is Model-Agnostic.
  10. Finding 3: Domain Expertise Predicts Accuracy; Political Orientation Does Not.
  11. Finding 4: Clustering Reveals Distinct Response Strategies.
  12. Finding 5: Initial Learning Effect Followed by Cognitive Fatigue.
  13. Discussion and Implications
  14. Limitations.
  15. Conclusion
  16. ...and 1 more sections

Figures (6)

  • Figure 1: The JudgeGPT interface. Participants rate each fragment on three continuous axes: source attribution, authenticity, and topic familiarity.
  • Figure 2: Source score (left) and authenticity score (right) distributions for machine- vs. human-written fragments. Overlapping distributions and a non-significant $t$-test indicate participants cannot distinguish the two conditions.
  • Figure 3: Mean source and authenticity scores per LLM ($\pm$ SE). The dashed line marks chance level (0.5). No model is reliably identified as machine-generated.
  • Figure 4: Relationships between participant covariates and judgment scores. Top: political view (weak slope). Bottom: fake news familiarity (positive slope). Regression lines with 95% CI shown.
  • Figure 5: Pairplot of participant-level mean scores colored by cluster. The two groups show clearly separated distributions on both axes.
  • ...and 1 more figures