FraudShield: Knowledge Graph Empowered Defense for LLMs against Fraud Attacks

TL;DR

FraudShield introduces a knowledge-graph–driven defense for LLMs against fraud, constructing a tactic–keyword graph to detect and evidence-fy fraudulent text and then highlighting key indicators with XML tags to guide secure responses. The method leverages a two-stage refinement of keyword associations and evidence selection, enabling a robust output $y = \arg\max_{y_i} \mathcal{M}(y_i \mid x', \mathcal{E})$ while preserving performance on benign inputs. Experimental results on the Fraud-R1 dataset across four LLMs and five fraud types show substantial improvements in Defense Success Rate (DSR) over baselines, with notable gains in single-round and multi-turn scenarios, and ablation confirms the critical roles of XML tagging and explicit evidence. Human studies and analysis underscore enhanced fraud awareness and interpretability, suggesting practical applicability to real-world AI systems, such as messaging apps and online services, with attention to ethical considerations and potential adversarial use.

Abstract

Large language models (LLMs) have been widely integrated into critical automated workflows, including contract review and job application processes. However, LLMs are susceptible to manipulation by fraudulent information, which can lead to harmful outcomes. Although advanced defense methods have been developed to address this issue, they often exhibit limitations in effectiveness, interpretability, and generalizability, particularly when applied to LLM-based applications. To address these challenges, we introduce FraudShield, a novel framework designed to protect LLMs from fraudulent content by leveraging a comprehensive analysis of fraud tactics. Specifically, FraudShield constructs and refines a fraud tactic-keyword knowledge graph to capture high-confidence associations between suspicious text and fraud techniques. The structured knowledge graph augments the original input by highlighting keywords and providing supporting evidence, guiding the LLM toward more secure responses. Extensive experiments show that FraudShield consistently outperforms state-of-the-art defenses across four mainstream LLMs and five representative fraud types, while also offering interpretable clues for the model's generations.

Figures (4)

• Figure 1: When encountering fraudulent job postings, both the vanilla LLM and the safety prompt–based defense fail to guard the model against fraudulent content, whereas FraudShield successfully identifies red flags and advises caution.
• Figure 2: FraudShield employs a tactic-driven and keyword-centric framework to counter fraudulent behaviors.
• Figure 3: Defense Success Rate@k of different LLMs and settings.
• Figure 4: Tactic counts, average confidence scores and distribution of fraud tactics.