Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Incentivizing News Consumption on Social Media Platforms Using Large Language Models and Realistic Bot Accounts

Hadi Askari, Anshuman Chhabra, Bernhard Clemm von Hohenberg, Michael Heseltine, Magdalena Wojcieszak

TL;DR

The study tackles the problem of pervasive under-exposure to quality news on social media and its implications for democratic discourse. It tests a scalable, bot-mediated nudging approach by deploying 28 GPT-2–based replies to 28,457 US Twitter users who discuss non-political topics, embedding links to verified, ideologically balanced outlets and prompts to follow those outlets. Using a validated outlet list (Ad Fontes), a BERT-based political content classifier, and entropy balancing to compare treated and control groups, the authors find only small, heterogeneous effects: increased following of news accounts for treated users and modest, gender-linked increases in liking news content, mainly among users with higher prior political interest. The results suggest limited efficacy of short, bot-driven nudges to shift on-platform news engagement, but point to potential cumulative effects and design considerations for future pro-social interventions on social platforms with stronger models and longer time horizons.

Abstract

Polarization, declining trust, and wavering support for democratic norms are pressing threats to U.S. democracy. Exposure to verified and quality news may lower individual susceptibility to these threats and make citizens more resilient to misinformation, populism, and hyperpartisan rhetoric. This project examines how to enhance users' exposure to and engagement with verified and ideologically balanced news in an ecologically valid setting. We rely on a large-scale two-week long field experiment (from 1/19/2023 to 2/3/2023) on 28,457 Twitter users. We created 28 bots utilizing GPT-2 that replied to users tweeting about sports, entertainment, or lifestyle with a contextual reply containing two hardcoded elements: a URL to the topic-relevant section of quality news organization and an encouragement to follow its Twitter account. To further test differential effects by gender of the bots, treated users were randomly assigned to receive responses by bots presented as female or male. We examine whether our over-time intervention enhances the following of news media organization, the sharing and the liking of news content and the tweeting about politics and the liking of political content. We find that the treated users followed more news accounts and the users in the female bot treatment were more likely to like news content than the control. Most of these results, however, were small in magnitude and confined to the already politically interested Twitter users, as indicated by their pre-treatment tweeting about politics. These findings have implications for social media and news organizations, and also offer direction for future work on how Large Language Models and other computational interventions can effectively enhance individual on-platform engagement with quality news and public affairs.

Incentivizing News Consumption on Social Media Platforms Using Large Language Models and Realistic Bot Accounts

TL;DR

The study tackles the problem of pervasive under-exposure to quality news on social media and its implications for democratic discourse. It tests a scalable, bot-mediated nudging approach by deploying 28 GPT-2–based replies to 28,457 US Twitter users who discuss non-political topics, embedding links to verified, ideologically balanced outlets and prompts to follow those outlets. Using a validated outlet list (Ad Fontes), a BERT-based political content classifier, and entropy balancing to compare treated and control groups, the authors find only small, heterogeneous effects: increased following of news accounts for treated users and modest, gender-linked increases in liking news content, mainly among users with higher prior political interest. The results suggest limited efficacy of short, bot-driven nudges to shift on-platform news engagement, but point to potential cumulative effects and design considerations for future pro-social interventions on social platforms with stronger models and longer time horizons.

Abstract

Polarization, declining trust, and wavering support for democratic norms are pressing threats to U.S. democracy. Exposure to verified and quality news may lower individual susceptibility to these threats and make citizens more resilient to misinformation, populism, and hyperpartisan rhetoric. This project examines how to enhance users' exposure to and engagement with verified and ideologically balanced news in an ecologically valid setting. We rely on a large-scale two-week long field experiment (from 1/19/2023 to 2/3/2023) on 28,457 Twitter users. We created 28 bots utilizing GPT-2 that replied to users tweeting about sports, entertainment, or lifestyle with a contextual reply containing two hardcoded elements: a URL to the topic-relevant section of quality news organization and an encouragement to follow its Twitter account. To further test differential effects by gender of the bots, treated users were randomly assigned to receive responses by bots presented as female or male. We examine whether our over-time intervention enhances the following of news media organization, the sharing and the liking of news content and the tweeting about politics and the liking of political content. We find that the treated users followed more news accounts and the users in the female bot treatment were more likely to like news content than the control. Most of these results, however, were small in magnitude and confined to the already politically interested Twitter users, as indicated by their pre-treatment tweeting about politics. These findings have implications for social media and news organizations, and also offer direction for future work on how Large Language Models and other computational interventions can effectively enhance individual on-platform engagement with quality news and public affairs.
Paper Structure (6 sections, 6 figures)

This paper contains 6 sections, 6 figures.

Table of Contents

  1. Introduction
  2. Data and Measurement
  3. Results
  4. Descriptives
  5. Treatment Effects
  6. Discussion

Figures (6)

  • Figure 1: Overview of the Experiment Design.
  • Figure 2: Sample Male and Female Bot Accounts.
  • Figure 3: User Distribution Across Pre-Treatment Measures. Followed news media accounts are a count measure based on all recorded accounts followed. News media likes, news media (re)tweets, political likes, and political (re)tweets are measured as a count based on the last 100 likes or (re)tweets made prior to the treatment period.
  • Figure 4: Main Effects Plot: Coefficient estimates and 95% confidence intervals for G-computation after entropy balancing regression models with robust standard errors. Dependent variables taken as the difference between pre- and post-treatment individual user measures. News media accounts followed measured as a count, news media (re)tweets and likes and political (re)tweets and likes measured as percentages.
  • Figure 5: Main Treatment Effects Divided by Users' Prior Political Engagement Levels:Coefficient estimates and 95% confidence intervals for G-computation after entropy balancing regression models with robust standard errors. Dependent variables taken as the difference between pre- and post-treatment individual user measures. News media accounts followed measured as a count, news media (re)tweets and likes and political (re)tweets and likes measured as percentages.
  • ...and 1 more figures