Modes of Analyzing Disinformation Narratives With AI/ML/Text Mining to Assist in Mitigating the Weaponization of Social Media
Andy Skumanich, Han Kyul Kim
TL;DR
The paper tackles the weaponization of social media through mal-info and proposes a Capture/Track/Respond (CTR) framework leveraging AI/ML text mining to quantify and monitor mal-info for human expert intervention. It analyzes fringe networks (Gab, Gettr, Bitchute), employing keyness analysis and TF-IDF to extract semi-quantitative signatures, and examines Gab's GenAI bot Conspiracy AI to illustrate the risks of generative amplification. The findings demonstrate the feasibility of deriving meaningful, time-evolving signatures from noisy fringe data and argue for scalable CTR tools to assist mitigation, while highlighting mal-GenAI threats and the need for guardrails. The work aims to provide a foundation for quick, data-driven monitoring that can inform policy, counter-messaging, and cross-platform analyses to reduce societal harm from online disinformation.
Abstract
This paper highlights the developing need for quantitative modes for capturing and monitoring malicious communication in social media. There has been a deliberate "weaponization" of messaging through the use of social networks including by politically oriented entities both state sponsored and privately run. The article identifies a use of AI/ML characterization of generalized "mal-info," a broad term which includes deliberate malicious narratives similar with hate speech, which adversely impact society. A key point of the discussion is that this mal-info will dramatically increase in volume, and it will become essential for sharable quantifying tools to provide support for human expert intervention. Despite attempts to introduce moderation on major platforms like Facebook and X/Twitter, there are now established alternative social networks that offer completely unmoderated spaces. The paper presents an introduction to these platforms and the initial results of a qualitative and semi-quantitative analysis of characteristic mal-info posts. The authors perform a rudimentary text mining function for a preliminary characterization in order to evaluate the modes for better-automated monitoring. The action examines several inflammatory terms using text analysis and, importantly, discusses the use of generative algorithms by one political agent in particular, providing some examples of the potential risks to society. This latter is of grave concern, and monitoring tools must be established. This paper presents a preliminary step to selecting relevant sources and to setting a foundation for characterizing the mal-info, which must be monitored. The AI/ML methods provide a means for semi-quantitative signature capture. The impending use of "mal-GenAI" is presented.