BotSim: LLM-Powered Malicious Social Botnet Simulation

TL;DR

BotSim addresses the threat of evolving LLM-powered social bots by delivering a scalable simulation framework and the BotSim-24 Reddit-based detection dataset. The framework models bot–human interactions through a four-component architecture (social environment, environment perception, action list, and agent decision center) operating over a timeline $T$ and interaction graph $D$ with user sets $U_H$ and $U_B$. BotSim-24 comprises 1,907 human accounts and 1,000 LLM-driven bot accounts across six SubReddits, employing disguise strategies across metadata, text, and interactions; experiments show that traditional detectors falter on BotSim-24, while graph-based detectors—especially heterogeneous graphs—offer stronger signals. The findings highlight the growing difficulty of detecting LLM-driven bots, the need for novel detection strategies, and the practical value of BotSim as a research platform for cybersecurity and OSN regulation.

Abstract

Social media platforms like X(Twitter) and Reddit are vital to global communication. However, advancements in Large Language Model (LLM) technology give rise to social media bots with unprecedented intelligence. These bots adeptly simulate human profiles, conversations, and interactions, disseminating large amounts of false information and posing significant challenges to platform regulation. To better understand and counter these threats, we innovatively design BotSim, a malicious social botnet simulation powered by LLM. BotSim mimics the information dissemination patterns of real-world social networks, creating a virtual environment composed of intelligent agent bots and real human users. In the temporal simulation constructed by BotSim, these advanced agent bots autonomously engage in social interactions such as posting and commenting, effectively modeling scenarios of information flow and user interaction. Building on the BotSim framework, we construct a highly human-like, LLM-driven bot dataset called BotSim-24 and benchmark multiple bot detection strategies against it. The experimental results indicate that detection methods effective on traditional bot datasets perform worse on BotSim-24, highlighting the urgent need for new detection strategies to address the cybersecurity threats posed by these advanced bots.

Figures (8)

• Figure 1: The overall framework of BotSim.
• Figure 2: The impact of different proportions of edge perturbations on RGCN and S-HGN detection performance.
• Figure 3: BotSim-24 bot-human interaction edge illustration. Blue nodes represent human users, red nodes represent bot users, and the edges in the graph indicate that a user's first-level comment comments on another user's posting.
• Figure 4: The performance of human annotators in bot identification. 'Acc' indicates 'Accuracy', 'Pre' indicates 'precision', and 'Rec' indicates 'Recall'
• Figure 5: Statistics of the relationship between the number of users' posts, the number of first-level comments, and the number of second-level comments and their action frequency