Generation Z's Ability to Discriminate Between AI-generated and Human-Authored Text on Discord
Dhruv Ramu, Rishab Jain, Aditya Jain
TL;DR
Gen Z's ability to distinguish AI-generated from human-authored text on Discord is limited. The study uses a controlled Discord-based task with $n=335$ Gen Z participants and a one-shot prompt to generate $50$ messages (split evenly between AI and human) across $25$ hobbies, evaluated via a Typeform survey. Findings show discrimination is above random but exhibits strong biases: AI-text identification varies with AI familiarity ($p<2.2\times10^{-16}$) while human-text identification improves among those with no AI familiarity; age and education positively correlate with accuracy, whereas gender and continent show weaker effects. These results have implications for AI literacy, trust in AI-enabled platforms, and the design of social-media interfaces that host AI-driven content.
Abstract
The growing popularity of generative artificial intelligence (AI) chatbots such as ChatGPT is having transformative effects on social media. As the prevalence of AI-generated content grows, concerns have been raised regarding privacy and misinformation online. Among social media platforms, Discord enables AI integrations -- making their primarily "Generation Z" userbase particularly exposed to AI-generated content. We surveyed Generation Z aged individuals (n = 335) to evaluate their proficiency in discriminating between AI-generated and human-authored text on Discord. The investigation employed one-shot prompting of ChatGPT, disguised as a text message received on the Discord.com platform. We explore the influence of demographic factors on ability, as well as participants' familiarity with Discord and artificial intelligence technologies. We find that Generation Z individuals are unable to discern between AI and human-authored text (p = 0.011), and that those with lower self-reported familiarity with Discord demonstrated an improved ability in identifying human-authored compared to those with self-reported experience with AI (p << 0.0001). Our results suggest that there is a nuanced relationship between AI technology and popular modes of communication for Generation Z, contributing valuable insights into human-computer interactions, digital communication, and artificial intelligence literacy.