Detecting LLM-Generated Spam Reviews by Integrating Language Model Embeddings and Graph Neural Network

TL;DR

The paper tackles the rising problem of LLM-generated spam reviews by synthesizing three realistic datasets and introducing FraudSquad, a hybrid detector that fuses LM-derived embeddings with a gated graph transformer on a review graph. FraudSquad achieves state-of-the-art detection without manual feature engineering and with minimal labeled data, outperforming baselines by up to 44.22% in precision and 43.01% in recall on synthetic datasets and showing strong results on human-written spam as well. The approach leverages both semantic content and user-behavior signals, demonstrating robustness across datasets and highlighting the importance of adapting spam detection to LLM-era threats. The work provides practical impact for online platforms by offering an efficient, scalable framework and releasing code and synthetic datasets to advance research in e-commerce security.

Abstract

The rise of large language models (LLMs) has enabled the generation of highly persuasive spam reviews that closely mimic human writing. These reviews pose significant challenges for existing detection systems and threaten the credibility of online platforms. In this work, we first create three realistic LLM-generated spam review datasets using three distinct LLMs, each guided by product metadata and genuine reference reviews. Evaluations by GPT-4.1 confirm the high persuasion and deceptive potential of these reviews. To address this threat, we propose FraudSquad, a hybrid detection model that integrates text embeddings from a pre-trained language model with a gated graph transformer for spam node classification. FraudSquad captures both semantic and behavioral signals without relying on manual feature engineering or massive training resources. Experiments show that FraudSquad outperforms state-of-the-art baselines by up to 44.22% in precision and 43.01% in recall on three LLM-generated datasets, while also achieving promising results on two human-written spam datasets. Furthermore, FraudSquad maintains a modest model size and requires minimal labeled training data, making it a practical solution for real-world applications. Our contributions include new synthetic datasets, a practical detection framework, and empirical evidence highlighting the urgency of adapting spam detection to the LLM era. Our code and datasets are available at: https://anonymous.4open.science/r/FraudSquad-5389/.

Figures (4)

• Figure 1: Workflow of LLM-generated review spamming. There are three steps. Step 1: Provide the LLM-based chatbot with the product metadata, genuine reference review texts, and specific output requirements. Step 2: Obtain the review texts generated by the LLM based on the input. Step 3: Use controlled accounts to post the generated reviews.
• Figure 2: Evaluation results of LLM-generated and human-written review texts by GPT-4.1, rated on a Likert scale. The results show that LLM-generated reviews outperform human-written ones in terms of being positive, detailed, convincing, and influential. Moreover, they are often perceived as highly human-like in style and tone.
• Figure 3: The overall architecture of FraudSquad by integrating LM-enhanced node embedding and graph neural network on the constructed review graph, where the review nodes are connected by various relations. The graph neural network in FraudSquad employs multi-headed gated graph transformer layers, which are also illustrated in this figure.
• Figure 4: Ablation study on node embeddings (different bars) and gated graph transformer (with vs. without sets of bars).