Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Who Writes the Review, Human or AI?

Panagiotis C. Theocharopoulos, Spiros V. Georgakopoulos, Sotiris K. Tasoulis, Vassilis P. Plagianakos

TL;DR

This work tackles the problem of detecting AI-generated text within book reviews by employing a transfer-learning framework that generalizes across topics. It builds a 20,000-entry dataset using Vicuna-generated reviews paired with real reviews and uses Word2Vec embeddings with an LSTM classifier, fine-tuned from a model trained on COVID-19 abstracts. The approach achieves an average accuracy of 96.9% and offers analysis of misclassifications and feature representations, highlighting cross-domain feasibility and remaining challenges. The findings have practical implications for maintaining content authenticity and guiding detection strategies as LLMs evolve.

Abstract

With the increasing use of Artificial Intelligence in Natural Language Processing, concerns have been raised regarding the detection of AI-generated text in various domains. This study aims to investigate this issue by proposing a methodology to accurately distinguish AI-generated and human-written book reviews. Our approach utilizes transfer learning, enabling the model to identify generated text across different topics while improving its ability to detect variations in writing style and vocabulary. To evaluate the effectiveness of the proposed methodology, we developed a dataset consisting of real book reviews and AI-generated reviews using the recently proposed Vicuna open-source language model. The experimental results demonstrate that it is feasible to detect the original source of text, achieving an accuracy rate of 96.86%. Our efforts are oriented toward the exploration of the capabilities and limitations of Large Language Models in the context of text identification. Expanding our knowledge in these aspects will be valuable for effectively navigating similar models in the future and ensuring the integrity and authenticity of human-generated content.

Who Writes the Review, Human or AI?

TL;DR

This work tackles the problem of detecting AI-generated text within book reviews by employing a transfer-learning framework that generalizes across topics. It builds a 20,000-entry dataset using Vicuna-generated reviews paired with real reviews and uses Word2Vec embeddings with an LSTM classifier, fine-tuned from a model trained on COVID-19 abstracts. The approach achieves an average accuracy of 96.9% and offers analysis of misclassifications and feature representations, highlighting cross-domain feasibility and remaining challenges. The findings have practical implications for maintaining content authenticity and guiding detection strategies as LLMs evolve.

Abstract

With the increasing use of Artificial Intelligence in Natural Language Processing, concerns have been raised regarding the detection of AI-generated text in various domains. This study aims to investigate this issue by proposing a methodology to accurately distinguish AI-generated and human-written book reviews. Our approach utilizes transfer learning, enabling the model to identify generated text across different topics while improving its ability to detect variations in writing style and vocabulary. To evaluate the effectiveness of the proposed methodology, we developed a dataset consisting of real book reviews and AI-generated reviews using the recently proposed Vicuna open-source language model. The experimental results demonstrate that it is feasible to detect the original source of text, achieving an accuracy rate of 96.86%. Our efforts are oriented toward the exploration of the capabilities and limitations of Large Language Models in the context of text identification. Expanding our knowledge in these aspects will be valuable for effectively navigating similar models in the future and ensuring the integrity and authenticity of human-generated content.
Paper Structure (8 sections, 5 figures, 2 tables)

This paper contains 8 sections, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Research Methodology
  4. AI-Generated Dataset
  5. Text representation
  6. Model Architecture
  7. Experimental Results
  8. Conclusion

Figures (5)

  • Figure 1: Schematic overview of this study. Titles and book reviews have been collected from the Kaggle Dataset. The titles of the selected books have been prompted to the Vicuna model, which returned the AI-written reviews based on their title. The study involved text cleaning and data representation as well as the model's results evaluation.
  • Figure 2: Frequencies of the top 20 words on both original and AI-generated reviews.
  • Figure 3: Wordcloud of the misclassified texts.
  • Figure 4: Frequencies of the misclassified words.
  • Figure 5: t-SNE Visualization of Hidden State Representation on the evaluation data. The blue points represent the original reviews and the red are the AI-generated text.