When AI Agents Collude Online: Financial Fraud Risks by Collaborative LLM Agents on Social Platforms

TL;DR

The paper investigates the risk of collective financial fraud in autonomous LLM-driven agent societies on social platforms. It introduces MultiAgentFraudBench, a large-scale, OASIS-based benchmark that simulates 28 fraud scenarios across public and private interactions, with the metrics $R_{ ext{conv}}$ and $R_{ ext{pop}}$ to quantify conversational success and broad harm. Key findings show that higher model capability correlates with greater fraud risk, safety alignments fail to generalize to fraud contexts, and collusion among malicious agents markedly amplifies harm, while mitigation via monitoring and collective resilience can reduce impact. The work provides practical insights for platform design and policy, highlighting the need for multi-layer defenses against emergent, collaborative AI-enabled fraud.

Abstract

In this work, we study the risks of collective financial fraud in large-scale multi-agent systems powered by large language model (LLM) agents. We investigate whether agents can collaborate in fraudulent behaviors, how such collaboration amplifies risks, and what factors influence fraud success. To support this research, we present MultiAgentFraudBench, a large-scale benchmark for simulating financial fraud scenarios based on realistic online interactions. The benchmark covers 28 typical online fraud scenarios, spanning the full fraud lifecycle across both public and private domains. We further analyze key factors affecting fraud success, including interaction depth, activity level, and fine-grained collaboration failure modes. Finally, we propose a series of mitigation strategies, including adding content-level warnings to fraudulent posts and dialogues, using LLMs as monitors to block potentially malicious agents, and fostering group resilience through information sharing at the societal level. Notably, we observe that malicious agents can adapt to environmental interventions. Our findings highlight the real-world risks of multi-agent financial fraud and suggest practical measures for mitigating them. Code is available at https://github.com/zheng977/MutiAgent4Fraud.

Figures (11)

• Figure 1: (left): a diagram of fraud activities on social media: multiple malicious actors targeting benign users. (middle): at each time step, the recommendation system distributes posts to users, and users react to the posts or to messages from other users; (right): examples of agents evolving and colluding, and the three levels of mitigation we propose.
• Figure 1: Fraud susceptibility rates (%) across model families in simulated adversarial scenarios. Benign baseline: Qwen-2.5-32B-Instruct. Agent ratio: 1:10 (malicious:benign). $R_{\text{pop}}$ and $R_{\text{conv}}$ represent population and conversion rates respectively.
• Figure 1: Distribution of fraud categories in the balanced dataset (2,800 posts across 28 subcategories).
• Figure 2: (left): a diagram of fraud activities on social media: multiple malicious actors targeting benign users. (middle): at each time step, the recommendation system distributes posts to users, and users react to the posts or to messages from other users; (right): examples of agents evolving and colluding, and the three levels of mitigation we propose.
• Figure 2: Example synthesized fraud posts from the curated dataset. These examples mimic realistic social-media posts, combining persuasive language, visual appeal, and fabricated testimonials to reproduce authentic deception patterns.