Do You Trust ChatGPT? -- Perceived Credibility of Human and AI-Generated Content

TL;DR

The study investigates how UI presentation and content origin (human vs AI-generated) affect perceived credibility. Using a 606-participant online survey and a four-factor credibility model (competence, trustworthiness, clarity, engagement), it finds that UI has minimal impact, while AI-generated text is perceived as clearer and more engaging yet does not increase perceived competence or trust. Within-subject comparisons show shorter reading times for AI-generated content without reducing accuracy on attention checks. The findings urge labeling of AI-generated content and enhanced media literacy to mitigate overreliance and misperception of AI outputs in real-world information environments.

Abstract

This paper examines how individuals perceive the credibility of content originating from human authors versus content generated by large language models, like the GPT language model family that powers ChatGPT, in different user interface versions. Surprisingly, our results demonstrate that regardless of the user interface presentation, participants tend to attribute similar levels of credibility. While participants also do not report any different perceptions of competence and trustworthiness between human and AI-generated content, they rate AI-generated content as being clearer and more engaging. The findings from this study serve as a call for a more discerning approach to evaluating information sources, encouraging users to exercise caution and critical thinking when engaging with content generated by AI systems.

Figures (3)

• Figure 1: Treatment Conditions - UI Versions, Topic of Example Text: Academy Awards, human-generated text version
• Figure 2: Boxplots of Descriptive Results - Credibility Perceptions per UI Condition and Content Source
• Figure 3: Boxplots of Descriptive Results - Credibility Perceptions per Content Source