Ghostbuster: Detecting Text Ghostwritten by Large Language Models

TL;DR

Ghostbuster tackles AI-generated text detection by leveraging probabilities from weaker language models, applying a structured search over feature combinations, and training a simple linear classifier. This design aims to generalize across domains, prompts, and models without requiring access to the target model’s token probabilities. The authors release three cross-domain benchmarks (news, creative writing, student essays) and demonstrate strong in-domain and cross-domain performance, outperforming prior detectors. They also probe robustness to perturbations and discuss ethical considerations, highlighting careful deployment in real-world settings.

Abstract

We introduce Ghostbuster, a state-of-the-art system for detecting AI-generated text. Our method works by passing documents through a series of weaker language models, running a structured search over possible combinations of their features, and then training a classifier on the selected features to predict whether documents are AI-generated. Crucially, Ghostbuster does not require access to token probabilities from the target model, making it useful for detecting text generated by black-box models or unknown model versions. In conjunction with our model, we release three new datasets of human- and AI-generated text as detection benchmarks in the domains of student essays, creative writing, and news articles. We compare Ghostbuster to a variety of existing detectors, including DetectGPT and GPTZero, as well as a new RoBERTa baseline. Ghostbuster achieves 99.0 F1 when evaluated across domains, which is 5.9 F1 higher than the best preexisting model. It also outperforms all previous approaches in generalization across writing domains (+7.5 F1), prompting strategies (+2.1 F1), and language models (+4.4 F1). We also analyze the robustness of our system to a variety of perturbations and paraphrasing attacks and evaluate its performance on documents written by non-native English speakers.

Figures (6)

• Figure 1: An outline of our model training procedure. First, we fed each document into a series of weaker language models to obtain token probabilities. Then, we ran a structured search over combinations of the model outputs and trained a linear classifier on the selected features.
• Figure 2: An example comparison between human-written and ChatGPT-written text. Whenever possible, all ChatGPT-generated documents were generated based on the same prompts seen by human writers. When prompts were not available, we used ChatGPT to first generate a prompt conditioned on the human-written text and then used ChatGPT again to generate a document conditioned on the generated prompt.
• Figure 3: Robustness experiments on Ghostbuster (F1). We evaluated the performance of our system on documents that underwent a number of character- and word-level perturbations (left) as well as sentence- and paragraph-level perturbations (right). We describe the details of these perturbations in \ref{['sec:robustness']}.
• Figure 4: Model performance across document lengths, for various domain shifts (F1). We evaluated our model on trimmed documents. Our model predictably performed better on longer documents, whether evaluated in-domain or out-of-domain. We note that Ghostbuster may be unreliable for documents with $\le 100$ tokens, and its performance levels off with $\ge 500$ tokens.
• Figure 5: A list of features selected when performing forward feature selection on each of the three domains. We show in \ref{['tab:results3']} that these features lead to significantly better performance than using randomly chosen features or classifying based on our set of manually-selected features alone.