MAiDE-up: Multilingual Deception Detection of GPT-generated Hotel Reviews

TL;DR

This work tackles the challenge of deception detection in hotel reviews generated by large language models across ten languages. It introduces MAiDE-up, a balanced 20,000-review dataset (10k real, 10k AI-generated) spanning language, location, and sentiment, and conducts extensive linguistic analyses (analytic writing, descriptiveness, readability, topic modeling) to contrast AI vs real reviews. The study benchmarks deception-detection models (Random, Naive Bayes, Random Forest, and XLM-RoBERTa) under default and few-shot settings, revealing that sentiment, location, and prompt-language significantly affect performance, with humans finding the task moderately difficult. The authors provide open access to the dataset and models, discuss multilingual limitations of current linguistic tools, and advocate for transparent, ethical use of detection technologies to combat AI-generated deception in consumer reviews.

Abstract

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and deceptive human reviews, much less is known about the distinction between real reviews and AI-authored fake reviews. Moreover, most of the research so far has focused primarily on English, with very little work dedicated to other languages. In this paper, we compile and make publicly available the MAiDE-up dataset, consisting of 10,000 real and 10,000 AI-generated fake hotel reviews, balanced across ten languages. Using this dataset, we conduct extensive linguistic analyses to (1) compare the AI fake hotel reviews to real hotel reviews, and (2) identify the factors that influence the deception detection model performance. We explore the effectiveness of several models for deception detection in hotel reviews across three main dimensions: sentiment, location, and language. We find that these dimensions influence how well we can detect AI-generated fake reviews.

Figures (9)

• Figure 1: An example of an English positive review, rated with a score of 10, that contains both "upside" and "downside" sections. The reviewer can choose to write in just one or both sections.
• Figure 2: Accuracy measured with XLM-RoBERTa and best Random Forest model on different ratios of training data. The accuracy plateaus at $~$10%, i.e., 2,000 reviews.
• Figure 3: Accuracy with XLM-RoBERTa model per (a) review language, (b) prompt language, and (c) hotel location on the few-shot train-test split.
• Figure 4: Human accuracy per review language.
• Figure 5: We use the language filter bar to select the language we specify, which is also automated by using Selenium in the data collection process.