Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How A/B testing changes the dynamics of information spreading on a social network

Matteo Ottaviani, Stefan M. Herzog, Pietro Leonardo Nickl, Philipp Lorenz-Spreen

TL;DR

The paper investigates how A/B testing alters information diffusion on social networks by combining linguistic feature extraction from Upworthy-style headlines with an agent-based diffusion model. It introduces a data-driven decision model using LASSO to map headline features to click-through rate, then contrasts pure social spreading with A/B testing–driven selection across various network topologies under both frequentist and Bayesian AB regimes. Key findings from pilot analyses indicate that A/B testing increases homogeneity in the final distribution of content, amplifying the most successful features and reducing exploration. The work provides a framework for evaluating and potentially mitigating unwanted effects of algorithmic testing on online discourse, with implications for policy and platform design to preserve user autonomy and a diverse information ecosystem.

Abstract

A/B testing methodology is generally performed by private companies to increase user engagement and satisfaction about online features. Their usage is far from being transparent and may undermine user autonomy (e.g. polarizing individual opinions, mis- and dis- information spreading). For our analysis we leverage a crucial case study dataset (i.e. Upworthy) where news headlines were allocated to users and reshuffled for optimizing clicks. Our centre of focus is to determine how and under which conditions A/B testing affects the distribution of content on the collective level, specifically on different social network structures. In order to achieve that, we set up an agent-based model reproducing social interaction and an individual decision-making model. Our preliminary results indicate that A/B testing has a substantial influence on the qualitative dynamics of information dissemination on a social network. Moreover, our modeling framework promisingly embeds conjecturing policy (e.g. nudging, boosting) interventions.

How A/B testing changes the dynamics of information spreading on a social network

TL;DR

The paper investigates how A/B testing alters information diffusion on social networks by combining linguistic feature extraction from Upworthy-style headlines with an agent-based diffusion model. It introduces a data-driven decision model using LASSO to map headline features to click-through rate, then contrasts pure social spreading with A/B testing–driven selection across various network topologies under both frequentist and Bayesian AB regimes. Key findings from pilot analyses indicate that A/B testing increases homogeneity in the final distribution of content, amplifying the most successful features and reducing exploration. The work provides a framework for evaluating and potentially mitigating unwanted effects of algorithmic testing on online discourse, with implications for policy and platform design to preserve user autonomy and a diverse information ecosystem.

Abstract

A/B testing methodology is generally performed by private companies to increase user engagement and satisfaction about online features. Their usage is far from being transparent and may undermine user autonomy (e.g. polarizing individual opinions, mis- and dis- information spreading). For our analysis we leverage a crucial case study dataset (i.e. Upworthy) where news headlines were allocated to users and reshuffled for optimizing clicks. Our centre of focus is to determine how and under which conditions A/B testing affects the distribution of content on the collective level, specifically on different social network structures. In order to achieve that, we set up an agent-based model reproducing social interaction and an individual decision-making model. Our preliminary results indicate that A/B testing has a substantial influence on the qualitative dynamics of information dissemination on a social network. Moreover, our modeling framework promisingly embeds conjecturing policy (e.g. nudging, boosting) interventions.
Paper Structure (23 sections, 5 equations, 4 figures, 1 table)

This paper contains 23 sections, 5 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. A/B testing: a brief overview
  3. A case of study: Upworthy.com
  4. The Upworthy Research Archive
  5. Clickbait: definition and characteristics, linguistic features
  6. Clickbait style
  7. Methods
  8. Linguistic features extraction
  9. Formal Features
  10. Semantic features
  11. Sentimental Analysis
  12. Topic model
  13. Assessing linguistic features importance in the data set
  14. Individual decision-making model
  15. LASSO
  16. ...and 8 more sections

Figures (4)

  • Figure 1: An example of package features from the Upworthy Research Archive, as described in matias2020asking.
  • Figure 2: Broad round-up of the temporal order of our procedure.
  • Figure 3: Comparison of the fist-ranked linguistic feature dynamics in both explored scenarios, a pure social spreading setting and an A/B testing led one. The network structure employed in the simulation is the Albert-Barabasi one.
  • Figure 4: Comparison of the final ranking distributions of messages in both explored scenarios, a pure social spreading setting and an A/B testing led one. The network structure employed in the simulation is the Albert-Barabasi one.