Quantifying the Vulnerabilities of the Online Public Square to Adversarial Manipulation Tactics

TL;DR

Online social platforms function as a public square but are vulnerable to coordinated inauthentic manipulation. The authors develop SimSoM, an agent-based diffusion model on an empirical follower network to quantify how bad actors spreading low-quality content affect the overall information quality, parameterizing tactics as infiltration $\gamma$, deception $\phi$, flooding $\theta$, and targeting. The results show that infiltration is the most damaging tactic, reducing average quality by more than half at moderate infiltration, and that hubs (high-degree nodes) amplify the impact; flooding and deception can further degrade quality, while targeting influentials can yield counterintuitive effects by localizing harm. The study also reconstructs exposure vs. reshare cascades, showing low-quality content often achieves larger exposure under high appeal and infiltration, underscoring the need for platform defenses such as stronger bot detection, diffusion caps, and user-accuracy nudges to bolster resilience. Overall, the work provides a quantitative framework to evaluate manipulation tactics and design mitigation strategies for safeguarding the online public square.

Abstract

Social media, seen by some as the modern public square, is vulnerable to manipulation. By controlling inauthentic accounts impersonating humans, malicious actors can amplify disinformation within target communities. The consequences of such operations are difficult to evaluate due to the challenges posed by collecting data and carrying out ethical experiments that would influence online communities. Here we use a social media model that simulates information diffusion in an empirical network to quantify the impacts of several adversarial manipulation tactics on the quality of content. We find that the presence of influential accounts, a hallmark of social media, exacerbates the vulnerabilities of online communities to manipulation. Among the explored tactics that bad actors can employ, infiltrating a community is the most likely to make low-quality content go viral. Such harm can be further compounded by inauthentic agents flooding the network with low-quality, yet appealing content, but is mitigated when bad actors focus on specific targets, such as influential or vulnerable individuals. These insights suggest countermeasures that platforms could employ to increase the resilience of social media users to manipulation.

Figures (13)

• Figure 1: Illustration of the SimSoM model. Each agent has a limited-size news feed, containing messages posted or reposted by friends. Dashed arrows represent follower links; messages propagate from agents to their followers along solid links. At each time step, an active agent (colored node) either posts a new message (here, m20) or reposts one of the existing messages in their feed, selected with probability proportional to their appeal $a$, social engagement $e$, and recency $r$ (here, m2 is selected). The message spreads to the node's followers and shows up on their feeds.
• Figure 2: Subnetworks modeling authentic accounts (purple nodes) and bad actors (yellow nodes). (a) Illustration of the follower link structure. Solid links indicate follower relations within each subnetwork. Both subnetworks have hub and clustering structure that mimics or derives from online social networks. Dashed links represent authentic accounts following bad actors, according to the infiltration parameter $\gamma$, which represents the probability that an authentic node follows any given bad actor. When $\gamma=0$ there is no infiltration and bad actors are isolated, therefore harmless; the opposite extreme $\gamma=1$ indicates complete infiltration, such that bad actors are followed by all authentic accounts. (b) Effects of bad-actor infiltration $\gamma$ on the quality of messages in synthetic networks with $10^3$ authentic agents and 100 inauthentic agents. For illustration purposes, both the authentic and inauthentic subnetworks in this panel are generated with the same method used for the inauthentic subnetworks in our experiments (see Methods). Node size represents the number of followers. The darker an authentic agent node, the lower the quality of messages in their feed.
• Figure 3: Impacts of different network structural features on the average information quality, relative to the scenario without bad actors. The original network ("hubs + clusters") is visualized along with shuffled networks in which links from the original network are rewired while preserving clusters, hubs, or neither ("random"). Node size and color represent, respectively, the number of followers of an account and their political leaning ranging from liberal to conservative (red to blue, see Methods). Yellow nodes are bad actors. Pairwise statistical significance is calculated using the Mann–Whitney U test (*** for $p < 10^{-3}$); only significant differences are reported.
• Figure 4: Effects of individual and combined tactics by bad actors on the system's message quality, relative to the scenario without bad actors. (a) Varying infiltration $\gamma$, without flooding ($\theta=1$) or deception ($\phi$=0). Shading represents 95% confidence intervals across runs in panels a--c. (b) Varying flooding $\theta$ with infiltration $\gamma=0.01$ and no deception ($\phi=0$). (c) Varying deception $\phi$ with infiltration $\gamma=0.01$ and no flooding ($\theta=1$). (d) Joint infiltration and flooding with no deception. (e) Joint infiltration and deception with no flooding. (f) Joint deception and flooding with infiltration $\gamma=0.01$.
• Figure 5: Complementary cumulative distributions of reshare cascade sizes for low- and high-quality content, generated by inauthentic and authentic agents, respectively. The plots are based on 10 simulations. (a) Effect of bad-actor infiltration $\gamma$, with no flooding ($\theta=1$) or deception ($\phi=0$). (b) Effect of flooding $\theta$, with low infiltration ($\gamma=10^{-3}$) and no deception ($\phi=0$). (c) Effect of deception $\phi$, with low infiltration ($\gamma=10^{-3}$) and no flooding ($\theta=1$).