Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

What Matters in Explanations: Towards Explainable Fake Review Detection Focusing on Transformers

Md Shajalal, Md Atabuzzaman, Alexander Boden, Gunnar Stevens, Delong Du

TL;DR

This paper tackles fake review detection on E-commerce platforms and the need for explanations that help users understand model decisions. It proposes two transformer-based detectors, DistilBERT and XLNet, and uses Layer-wise Relevance Propagation to generate word-level explanations of predictions. The models achieve state-of-the-art performance on Fake Review and Yelp datasets, and an empirical user study with 12 participants assesses the usefulness of the explanations and identifies requirements for presentation. The work advances explainable NLP in fraud detection and informs future design of transparent detection systems.

Abstract

Customers' reviews and feedback play crucial role on electronic commerce~(E-commerce) platforms like Amazon, Zalando, and eBay in influencing other customers' purchasing decisions. However, there is a prevailing concern that sellers often post fake or spam reviews to deceive potential customers and manipulate their opinions about a product. Over the past decade, there has been considerable interest in using machine learning (ML) and deep learning (DL) models to identify such fraudulent reviews. Unfortunately, the decisions made by complex ML and DL models - which often function as \emph{black-boxes} - can be surprising and difficult for general users to comprehend. In this paper, we propose an explainable framework for detecting fake reviews with high precision in identifying fraudulent content with explanations and investigate what information matters most for explaining particular decisions by conducting empirical user evaluation. Initially, we develop fake review detection models using DL and transformer models including XLNet and DistilBERT. We then introduce layer-wise relevance propagation (LRP) technique for generating explanations that can map the contributions of words toward the predicted class. The experimental results on two benchmark fake review detection datasets demonstrate that our predictive models achieve state-of-the-art performance and outperform several existing methods. Furthermore, the empirical user evaluation of the generated explanations concludes which important information needs to be considered in generating explanations in the context of fake review identification.

What Matters in Explanations: Towards Explainable Fake Review Detection Focusing on Transformers

TL;DR

This paper tackles fake review detection on E-commerce platforms and the need for explanations that help users understand model decisions. It proposes two transformer-based detectors, DistilBERT and XLNet, and uses Layer-wise Relevance Propagation to generate word-level explanations of predictions. The models achieve state-of-the-art performance on Fake Review and Yelp datasets, and an empirical user study with 12 participants assesses the usefulness of the explanations and identifies requirements for presentation. The work advances explainable NLP in fraud detection and informs future design of transparent detection systems.

Abstract

Customers' reviews and feedback play crucial role on electronic commerce~(E-commerce) platforms like Amazon, Zalando, and eBay in influencing other customers' purchasing decisions. However, there is a prevailing concern that sellers often post fake or spam reviews to deceive potential customers and manipulate their opinions about a product. Over the past decade, there has been considerable interest in using machine learning (ML) and deep learning (DL) models to identify such fraudulent reviews. Unfortunately, the decisions made by complex ML and DL models - which often function as \emph{black-boxes} - can be surprising and difficult for general users to comprehend. In this paper, we propose an explainable framework for detecting fake reviews with high precision in identifying fraudulent content with explanations and investigate what information matters most for explaining particular decisions by conducting empirical user evaluation. Initially, we develop fake review detection models using DL and transformer models including XLNet and DistilBERT. We then introduce layer-wise relevance propagation (LRP) technique for generating explanations that can map the contributions of words toward the predicted class. The experimental results on two benchmark fake review detection datasets demonstrate that our predictive models achieve state-of-the-art performance and outperform several existing methods. Furthermore, the empirical user evaluation of the generated explanations concludes which important information needs to be considered in generating explanations in the context of fake review identification.
Paper Structure (15 sections, 1 equation, 4 figures, 5 tables)

This paper contains 15 sections, 1 equation, 4 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Literature Review
  3. Our Method
  4. DistilBERT Transformer
  5. XLNet Transformer
  6. Explaining the prediction
  7. Experiments
  8. Dataset
  9. Experimental Settings
  10. Experimental Results
  11. Performance on Fake Review Dataset
  12. Performance on Yelp Review Dataset:
  13. Explainability of the predictions
  14. Empirical User Evaluation
  15. Conclusion and Future Direction

Figures (4)

  • Figure 1: Explanation with highlighting relevant words for a predicted fake review.
  • Figure 2: Explanation with highlighting relevant words for a predicted fake review.
  • Figure 3: Explanation with highlighting relevant words for a predicted fake review for Yelp Dataset.
  • Figure 4: Explanation with highlighting relevant words for a predicted fake review for Yelp Dataset.