Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams

TL;DR

Romance-baiting scams operate as large, text-based criminal enterprises with Hook-Line-Sinker structures and labor hierarchies. The study integrates insider interviews and a blinded 7-day experiment to evaluate whether LLMs can automate core scam stages and outperform humans in building trust and eliciting compliance, revealing significant automation potential. It also critically assesses current safeguards, showing AI self-disclosure mechanisms and content filters fail to detect or deter such misuse, underscoring an urgent need for long-horizon detection and policy responses. Collectively, the work highlights a substantive risk of scalable, AI-assisted romance-baiting and calls for multifaceted defenses that address both technological and human-rights dimensions.

Abstract

Romance-baiting scams have become a major source of financial and emotional harm worldwide. These operations are run by organized crime syndicates that traffic thousands of people into forced labor, requiring them to build emotional intimacy with victims over weeks of text conversations before pressuring them into fraudulent cryptocurrency investments. Because the scams are inherently text-based, they raise urgent questions about the role of Large Language Models (LLMs) in both current and future automation. We investigate this intersection by interviewing 145 insiders and 5 scam victims, performing a blinded long-term conversation study comparing LLM scam agents to human operators, and executing an evaluation of commercial safety filters. Our findings show that LLMs are already widely deployed within scam organizations, with 87% of scam labor consisting of systematized conversational tasks readily susceptible to automation. In a week-long study, an LLM agent not only elicited greater trust from study participants (p=0.007) but also achieved higher compliance with requests than human operators (46% vs. 18% for humans). Meanwhile, popular safety filters detected 0.0% of romance baiting dialogues. Together, these results suggest that romance-baiting scams may be amenable to full-scale LLM automation, while existing defenses remain inadequate to prevent their expansion.

Figures (6)

• Figure 1: The three stages of a romance-baiting scam which we refer to as Hook, Line and Sinker. The illustration is depicted using genuine messages obtained from our interviewed victims. Our investigation explores how much of these scams are and will be automated using LLMs.
• Figure 2: The romance-baiting life-cycle. The Hook stage involves mass messaging and early filtering. The Line stage builds trust and a persona of success, often with multiple operators. The Sinker stage pressures victims into investing in fraudulent platforms, leading to major losses.
• Figure 3: LLM agent designed for our study. At the start of each day, a new dialogue prompt is created with the persona, instructions, daily agenda and a condensed dialogue history including the last 20 chats (top right). During the day the Dialogue module responds to incoming messages, decides whether to add an emoji, and generates a response using the dialogue prompt (top right). The response passes through a Humanization Pipeline that makes the response more naturalistic. After long periods of silence, the Re-engagement module tries to initiate up new conversation but only when contextually warranted. At the start of each day, the Maintenance module summarizes chats, archives history, and loads the next agenda. The icons indicate LLM calls.
• Figure 4: Comparison of trust scores between LLM and Human partners. The numbers in red indicate the p-values and effect sizes (r) from paired t-tests comparing trust scores
• Figure 5: Distribution of trust scores comparing baseline Interpersonal Trust with 'Overall Trust' towards human and LLM partners.