Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction

Andrea Cristina McGlinchey, Peter J Barclay

TL;DR

The paper addresses the risk of AI-generated creative fiction by developing lightweight ML detectors that distinguish human-written from machine-generated short excerpts of detective novels. It demonstrates that simple classifiers, notably Naïve Bayes and a tuned MLP, can achieve accuracies above 95% on ~100-word samples, outperforming human judges by a wide margin. An online tool, AI Detective, is deployed to facilitate practical use in editorial workflows. The work highlights strong generalization to unseen novels within the same genre and outlines clear directions for expanding robustness and applicability across styles and longer texts.

Abstract

Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.

Using Machine Learning to Distinguish Human-written from Machine-generated Creative Fiction

TL;DR

The paper addresses the risk of AI-generated creative fiction by developing lightweight ML detectors that distinguish human-written from machine-generated short excerpts of detective novels. It demonstrates that simple classifiers, notably Naïve Bayes and a tuned MLP, can achieve accuracies above 95% on ~100-word samples, outperforming human judges by a wide margin. An online tool, AI Detective, is deployed to facilitate practical use in editorial workflows. The work highlights strong generalization to unseen novels within the same genre and outlines clear directions for expanding robustness and applicability across styles and longer texts.

Abstract

Following the universal availability of generative AI systems with the release of ChatGPT, automatic detection of deceptive text created by Large Language Models has focused on domains such as academic plagiarism and "fake news". However, generative AI also poses a threat to the livelihood of creative writers, and perhaps to literary culture in general, through reduction in quality of published material. Training a Large Language Model on writers' output to generate "sham books" in a particular style seems to constitute a new form of plagiarism. This problem has been little researched. In this study, we trained Machine Learning classifier models to distinguish short samples of human-written from machine-generated creative fiction, focusing on classic detective novels. Our results show that a Naive Bayes and a Multi-Layer Perceptron classifier achieved a high degree of success (accuracy > 95%), significantly outperforming human judges (accuracy < 55%). This approach worked well with short text samples (around 100 words), which previous research has shown to be difficult to classify. We have deployed an online proof-of-concept classifier tool, AI Detective, as a first step towards developing lightweight and reliable applications for use by editors and publishers, with the aim of protecting the economic and cultural contribution of human authors.

Paper Structure

This paper contains 26 sections, 2 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. BACKGROUND
  3. Advances in Text Generation
  4. Text Generation -- Concerns for Misuse
  5. Related Work
  6. Detection of Automatically Generated Creative Fiction
  7. Aims of this Research
  8. EXPERIMENTAL METHOD
  9. Datasets and Data Preparation
  10. OpenAI API Settings
  11. Text Excerpt Length Distribution
  12. Models Tested
  13. Experimental Runs
  14. EXPERIMENTAL RESULTS
  15. Model Comparison
  16. ...and 11 more sections

Figures (2)

  • Figure 1: Distribution of character lengths for Human-written and AI-generated text.
  • Figure 2: Distribution of points scored on a Human vs. AI quiz