Characteristics and prevalence of fake social media profiles with AI-generated faces

TL;DR

The paper tackles the lack of empirical evidence on fake social media profiles that use AI-generated faces. It introduces the TwitterGAN dataset (1,420 accounts) and a light yet effective detection approach based on the consistent eye placement of GAN faces, complemented by human annotation, with prevalence estimated from a 1% random Twitter sample. It further validates and anchors its analysis with the AcademicGAN dataset and provides practical heuristics, releasing code and data to enable broader investigation. The findings reveal that GAN-generated profiles participate in scams, spam, and coordinated amplification, highlighting emerging threats to online integrity and motivating calls for improved detection, regulation, and AI literacy.

Abstract

Recent advancements in generative artificial intelligence (AI) have raised concerns about their potential to create convincing fake social media accounts, but empirical evidence is lacking. In this paper, we present a systematic analysis of Twitter (X) accounts using human faces generated by Generative Adversarial Networks (GANs) for their profile pictures. We present a dataset of 1,420 such accounts and show that they are used to spread scams, spam, and amplify coordinated messages, among other inauthentic activities. Leveraging a feature of GAN-generated faces -- consistent eye placement -- and supplementing it with human annotation, we devise an effective method for identifying GAN-generated profiles in the wild. Applying this method to a random sample of active Twitter users, we estimate a lower bound for the prevalence of profiles using GAN-generated faces between 0.021% and 0.044% -- around 10K daily active accounts. These findings underscore the emerging threats posed by multimodal generative AI. We release the source code of our detection method and the data we collect to facilitate further investigation. Additionally, we provide practical heuristics to assist social media users in recognizing such accounts.

Figures (6)

• Figure 1: Sample profile pictures from (a) AcademicGAN, (b) TwitterGAN, and (c) RandomTwitter.
• Figure 2: Profile characteristics of accounts in TwitterGAN. We show the distributions of (a) follower count, (b) following (friend) count, (c) tweet count, and (e) year of creation for astroturf and other accounts.
• Figure 3: Illustrations of the tactics and inauthentic activities of accounts in TwitterGAN. (a) Fake accounts impersonating persons that do not exist through GAN-generated profiles, human-like names, and fake descriptions. (b) Accounts replying fake screenshots with scam messages to other accounts (the keywords are redacted to avoid further spreading the scams). (c) Accounts posting spam messages repeatedly. (d) Accounts amplifying the messages from certain accounts by coordinately reposting their tweets. (e) Accounts leveraging ChatGPT to produce human-like replies. The account illustrations colored in red and blue represent inauthentic and normal ones, respectively.
• Figure 4: Common distinctive defects of GAN-generated faces that can help identify them. (a) Unreal hat. (b) Glass frame blended into the face; surreal background. (c) Unreal collar. (d) Earrings blended into the ears; irregular ear.
• Figure 5: Kernel density estimation distribution of the detected left and right eyes in (a) AcademicGAN, (b) TwitterGAN, and (c) a random sample of 25,000 accounts from RandomTwitter.