Efficient Detection of LLM-generated Texts with a Bayesian Surrogate Model

TL;DR

The paper tackles the problem of efficiently detecting LLM-generated text in a zero-shot setting. It introduces a Bayesian surrogate model based on Gaussian processes with a text-aware kernel to identify a small set of typical perturbations and interpolate scores to nearby samples, reducing the number of required queries to the source LLM. By sequentially selecting samples with the highest predictive uncertainty and using the GP to estimate the detection statistic $\ell({\bm{x}}, p_{\bm{\theta}}, q)$, the approach achieves substantial improvements in AUROC under low query budgets, outperforming DetectGPT on LLaMA2, Vicuna, and GPT-2 across multiple datasets. The method remains effective with proxy models for black-box LLMs like ChatGPT, demonstrating practical utility for real-world deployment and offering a foundation for applying Bayesian uncertainty-driven sample selection to other unsupervised detection tasks.

Abstract

The detection of machine-generated text, especially from large language models (LLMs), is crucial in preventing serious social problems resulting from their misuse. Some methods train dedicated detectors on specific datasets but fall short in generalizing to unseen test data, while other zero-shot ones often yield suboptimal performance. Although the recent DetectGPT has shown promising detection performance, it suffers from significant inefficiency issues, as detecting a single candidate requires querying the source LLM with hundreds of its perturbations. This paper aims to bridge this gap. Concretely, we propose to incorporate a Bayesian surrogate model, which allows us to select typical samples based on Bayesian uncertainty and interpolate scores from typical samples to other samples, to improve query efficiency. Empirical results demonstrate that our method significantly outperforms existing approaches under a low query budget. Notably, when detecting the text generated by LLaMA family models, our method with just 2 or 3 queries can outperform DetectGPT with 200 queries.

Figures (8)

• Figure 1: Method overview. Following DetectGPT mitchell2023detectgpt, we explore the local structure of the probability curvature of the LLM $p_{\bm{\theta}}$ to determine whether a text passage ${\bm{x}}$ originates from it. However, instead of using the source LLM to score numerous random perturbations, we leverage the high redundancy among these perturbations to enhance query efficiency. We select a limited number of typical samples for scoring and interpolate their scores to other samples. To achieve reasonable selection and interpolation, we employ a Gaussian process as the surrogate model, which, as shown, enjoys non-parametric flexibility and delivers calibrated uncertainty in the presence of a suitable kernel. The figure above also demonstrates the sequential selection of typical samples---at each step, the sample that the surrogate model is most uncertain about is chosen. After fitting, we use the surrogate model as a substitute for $\log p_{\bm{\theta}}$ to calculate the detection measure $\ell({\bm{x}}, p_{\bm{\theta}}, q)$ in \ref{['eq:detect-measure']}.
• Figure 2: The AUROC for detecting samples generated by GPT-2 varies depending on the number of queries made to the source GPT-2. We present the results on three representative datasets.
• Figure 3: The AUROC for detecting samples generated by GPT-2 varies depending on the number of queries made to the source GPT-2. We use T5-3B as the perturbation model here.
• Figure 4: Cross evaluation of using various source models (i.e., those generating the texts) and proxy models (i.e., those scoring the texts for detection) in detection. We select models from $\{$GPT-J, GPT-Neo-2.7, GPT-2$\}$. We report the average AUROC over the three datasets. We offer the row/column mean. The query budget is $15$.
• Figure 5: The visualization of the candidate text passage and the first $11$ typical perturbations of it identified by our method, ordered from top to bottom. The BertScore among them and the row mean (estimated without the diagonal elements) are reported. The log probabilities are given by GPT-2.