AI model GPT-3 (dis)informs us better than humans

TL;DR

The results of this preregistered study show that GPT-3 is a double-edge sword: in comparison with humans, it can produce accurate information that is easier to understand, but it can also produce more compelling disinformation.

Abstract

Artificial intelligence is changing the way we create and evaluate information, and this is happening during an infodemic, which has been having dramatic effects on global health. In this paper we evaluate whether recruited individuals can distinguish disinformation from accurate information, structured in the form of tweets, and determine whether a tweet is organic or synthetic, i.e., whether it has been written by a Twitter user or by the AI model GPT-3. Our results show that GPT-3 is a double-edge sword, which, in comparison with humans, can produce accurate information that is easier to understand, but can also produce more compelling disinformation. We also show that humans cannot distinguish tweets generated by GPT-3 from tweets written by human users. Starting from our results, we reflect on the dangers of AI for disinformation, and on how we can improve information campaigns to benefit global health.

Figures (12)

• Figure 1: GPT-3 AI model informs and disinform us better. Study set-up: GPT-3 was provided instruction prompts to produce synthetic tweets containing accurate versus inaccurate information (information vs disinformation). Through a Twitter search, organic tweets from Twitter users were retrieved and classified as accurate information or disinformation. The sum of synthetic and organic tweets, either containing information or disinformation, constitutes the study dataset. Human respondents were asked to recognize whether such tweets were true or false (i.e., accurate information or disinformation) and whether the tweets were organic (i.e., generated by Twitter users) or synthetic (i.e., generated by GPT-3). GPT-3 was asked to recognize whether the tweets were true or false (i.e., accurate information or disinformation) (A). Data collection: we gathered 869 responses to our survey. 157 responses were incomplete and were removed. Of 712 remaining responses, 15 were removed as they were completed too fast to be reliable. Our analysis was conducted on 697 complete and reliable responses (B). GPT-3 produces accurate and disinformation tweets that are recognized by human respondents as accurate more often than accurate and disinformation tweets, respectively, produced by humans. "Organic true" tweets (green column bars) are accurate tweets generated by Twitter users; "Synthetic true" (dotted green column bars) tweets are accurate tweets generated by GPT-3; "Organic false" (red column bars) tweets are disinformation tweets generated by Twitter users; "Synthetic false" (dotted red column bars) tweets are disinformation tweets generated by GPT-3 (C). Disinformation tweets (red column bars) are recognized more often correctly when compared with accurate tweets (green column bars). Synthetic tweets (dotted grey column bars) are recognized more often correctly when compared with organic tweets (grey column bars). Disinformation recognition score (or TF score) (0-1) is the average score for all 697 respondents ($1=100 \%$ correct answers; $0=0 \%$ correct answers); Ordinary one-way ANOVA multiple-comparisons Tukey's test, $\mathrm{n}=697, * * \mathrm{p}<0.01$;
• Figure 2: Humans evaluate information and disinformation better than GPT-3, and GPT-3 can "disobey" requests to generate disinformation. Green column bars represent successful responses given by human respondents, whereas green dotted bars represent successful responses given by GPT-3. Red bars represent incorrect responses from human respondents, whereas red dotted bars represent incorrect responses from GPT-3. The success rate concerning the evaluation of disinformation is$89 \%$ and $92 \%$ for GPT-3 and human respondents, respectively. The success rate concerning the evaluation of accurate information is $64 \%$ and $72 \%$ for GPT-3 and human respondents, respectively. The evaluation was conducted on organic tweets retrieved from Twitter which were included in our survey (A). Rate of "obedience" for GPT-3 - i.e., how often GPT-3 respected our request to generate information or disinformation tweets. For accurate information tweets, GPT-3 "obeyed" our request 99 times out of 101 requests, whereas for disinformation tweets, it "obeyed" our request 80 times out of 102 requests (B).
• Figure 3: A
• Figure 4: A
• Figure 5: The Confidence in recognizing disinformation increases post-survey, whereas the confidence in recognizing AI-generated information decreases; and proposed model to launch information campaigns and evaluate information. Respondents were asked to provide a score of how confident they were in their ability to recognize disinformation tweets before taking the survey (grey bar), and after taking the survey (black bar). Participants' confidence in disinformation recognition increased significantly from 3.05 to 3.49 out of 5. n=697; Welch's t-test, ****p<0.0001. Bars represent SEM (A). Respondents were asked to provide a score of how confident they were in their ability to recognize whether tweets were generated by humans (grey bar) or by AI (black bar). Participant's confidence in AI recognition dropped significantly from 2.69 to 1.7 out of 5 .$\mathrm{n}=697$; Welch's t-test, **** $\mathrm{p}<0.0001$. Bars represent SEM (B). Model for an efficient and inefficient communication strategy and launch of information campaign. Based on our data, and with the AI model adopted for our analysis, an efficient system relies on accurate information generated by GPT-3 (initiation phase), whereas it relies on trained humans to evaluate whether a piece of information is accurate or whether it contains disinformation (evaluation phase) (C). An inefficient system relies on humans to generate information and initiate an information campaign, and it relies on AI to evaluate whether a piece of information is accurate or whether it contains disinformation ( $\mathbf{C}^{\prime}$ ).