Distinguishing Chatbot from Human

TL;DR

The paper tackles the challenge of distinguishing human-written from GPT-generated text by assembling a large paired WikiHow-based dataset and evaluating two differing ML paradigms: feature-based classification using linguistic, structural, semantic, and interaction features, and embeddings-based classification with TFIDF, Word2Vec, GloVe, and BERT representations. It systematically analyzes model choices (LR, RF, SVM, XGBoost, MLP, DNN, LSTM), feature selection techniques (PCA, LDA, RF importance, Lasso), and embedding-based pipelines, reporting high accuracies (notably >0.96 and approaching or exceeding 0.99 in some settings). An ablation study pinpoints the lowercase_letter_ratio as a critical feature, with perturbations able to dramatically affect detectability, underscoring potential avenues for adversaries to evade detection. The results demonstrate robust discrimination capabilities and offer practical insights for textual analysis, moderation, and understanding the current state and limitations of chatbots in generating human-like content, while outlining future work to broaden model types, datasets, and post-processing strategies.

Abstract

There have been many recent advances in the fields of generative Artificial Intelligence (AI) and Large Language Models (LLM), with the Generative Pre-trained Transformer (GPT) model being a leading "chatbot." LLM-based chatbots have become so powerful that it may seem difficult to differentiate between human-written and machine-generated text. To analyze this problem, we have developed a new dataset consisting of more than 750,000 human-written paragraphs, with a corresponding chatbot-generated paragraph for each. Based on this dataset, we apply Machine Learning (ML) techniques to determine the origin of text (human or chatbot). Specifically, we consider two methodologies for tackling this issue: feature analysis and embeddings. Our feature analysis approach involves extracting a collection of features from the text for classification. We also explore the use of contextual embeddings and transformer-based architectures to train classification models. Our proposed solutions offer high classification accuracy and serve as useful tools for textual analysis, resulting in a better understanding of chatbot-generated text in this era of advanced AI technology.

Figures (42)

• Figure 1: ChatGPT prompt
• Figure 2: Distribution of number of words in paragraphs
• Figure 3: Correlation heatmap
• Figure 4: Random Forest feature importances
• Figure 5: Lasso feature importances